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DESCRIPTION 

SIGNAL PROCESSING DEVICE AND SIGNAL PROCESSING METHOD, AND 

PROGRAM AND RECORDING MEDIUM 

Technical Field 

The present invention relates to a signal processing 
device and signal processing method, and a program and 
recording medium, and in particular relates to a signal 
processing device and signal processing method, and a 
program and recording medium, enabling images and the like 
with closer approximation to real world signals. . 

Background Art 

Technology for detecting phenomena in the actual world 
(real world) with sensors and processing sampling data • 
output from the sensors is widely used. For example, image 
processing technology wherein the actual world is imaged 
with an imaging sensor and sampling data which is the image 
data is processed, is widely employed. 

Also, arrangements are known having second dimensions 
with fewer dimensions than first dimensions obtained by 
detecting with sensors first signals, which are signals of 
the real world having first dimensions, obtaining second 
signals including distortion as to the first signals, and 
performing signal processing based on the second signals, 
thereby generating third signals with alleviated distortion 
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as compared to the second signals (e.g., Japanese Unexamined 
Patent Application Publication No. 2001-250119) . 

However, conventionally, signal processing taking into 
consideration the continuity of real world signals had not 
been performed, so obtaining images and the like which 
closer approximate real world signals has been difficult. 
Disclosure of Invention 

The present invention has been made in light of the 
situation as described above, and provides for obtaining of 
images and the like which closer approximate real world 
signals . 

A signal processing device according to the present 
invention comprises: processing region setting means for 
setting the processing region within data, wherein light 
signals of the real world have been projected on a plurality 
of pixels each having time integration effects and a part of 
the continuity of the real world light signals has been 
lost; movement vector setting means for setting movement 
vectors of an object within the image data corresponding to 
the continuity of the real world light signals of which a 
part of the continuity has been lost in the image data; 
model generating means for modeling the region between pixel 
values of each of the pixels within the processing region 
and pixel values of each of the pixels where there is no 
movement blurring, with the understanding that the pixel 
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values of each of the pixels within the processing region 
are values obtained by pixel values of each of the pixels 
without movement blurring as to the object being integrated 
while moving corresponding to a movement vector; normal 
equation generating means for generating normal equations 
from a first equation wherein the pixel values of the pixels 
within the processing region have been substituted into a 
model generated by the model generating means, and a second 
equation for constraining the relation between the pixels 
where there is no movement blurring; and actual world 
estimating means for estimating the pixel values of each of 
the pixels where there is no movement blurring, by computing 
the normal equations generated from the normal equation 
generating means. 

The normal equation generating means allow a normal 
equation to be generated from a first equation wherein the 
pixel values of each of the pixels within the processing 
region are substituted as to the model generated by the 
model generating means, and a second equation wherein the 
difference of the pixel value between each of the pixels not 
generating movement blurring is zero. 

A signal processing method according to the present 
invention comprises: a processing region setting step for 
setting the processing region within data, wherein light 
signals of the real world have been projected on a plurality 
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of pixels each having time integration effects and a part of 
the continuity of the real world light signals has been 
lost; a movement vector setting step for setting movement 
vectors of an object within the image data corresponding to 
5 the continuity of the real world light signals of which a 
part of the continuity has been lost in the image data; a 
model generating step for modeling the region between pixel 
values of each of the pixels within the processing region 
and pixel values of each of the pixels where there is no 

10 movement blurring, with the understanding that the pixel 
values of each of the pixels within the processing region 
are values obtained by pixel values of each of the pixels 
without movement blurring as to the object being integrated 
while moving corresponding to a movement vector; a normal 

15- equation generating step for generating normal equations 

from a first equation wherein the pixel values of the pixels 
within the processing region have been substituted into a 
model generated in the model generating step, and a second 
equation for constraining the relation between the pixels 

2 0 where there is no movement blurring; and an actual world 

estimating step for estimating the pixel values of each of 
the pixels where there is no movement blurring, by computing 
the normal equations generated from the normal equation 
generating step . 

2 5 A program for a recording medium according to the 



- 5 - 

S04P0850 

present invention causes a computer to execute: a processing 
region setting step for setting the processing region within 
data, wherein light signals of the real world have been 
projected on a plurality of pixels each having time 
5 integration effects and a part of the continuity of the real 
world light signals has been lost; a movement vector setting 
step for setting movement vectors of an object within the 
image data corresponding to the continuity of the real world 
light signals of which a part of the continuity has been 

10 lost in the image data; a model generating step for modeling 
the region between pixel values of each of the pixels within 
the processing region and pixel values of each of the pixels 
where there is no movement blurring, with the understanding 
that the pixel values of each of the pixels within the 

15 processing region are values obtained by pixel values of 
each of the pixels without movement blurring as to the 
object being integrated while moving corresponding to a 
movement vector; a normal equation generating step for 
generating normal equations from a first equation wherein 

20 the pixel values of the pixels within the processing region 
have been substituted into a model generated in the model 
generating step, and a second equation for constraining the 
relation between the pixels where there is no movement 
blurring; and an actual world estimating step for estimating 

25 the pixel values of each of the pixels where there is no 
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movement blurring, by computing the normal equations 
generated from the normal equation generating step. 

A program according to the present invention causes a 
computer to execute: a processing region setting step for 
setting the processing region within data, wherein light 
signals of the real world have been projected on a plurality 
of pixels each having time integration effects and a part of 
the continuity of the real world light signals has been 
lost; a movement vector setting step for setting movement 
vectors of an object within the image data corresponding to 
the continuity of the real world light signals of which a 
part of the continuity has been lost in the image data; a 
model generating step for modeling the region between pixel 
values of each of the pixels within the processing region 
and pixel values of each of the pixels where there is no 
movement blurring, with the understanding that the pixel 
values of each of the pixels within the processing region 
are values obtained by pixel values of each of the pixels 
without movement blurring as to the object being integrated 
while moving corresponding to a movement vector; a normal 
equation generating step for generating normal equations 
from a first equation wherein the pixel values of the pixels 
within the processing region have been substituted into a 
model generated in the model generating step, and a second 
equation for constraining the relation between the pixels 
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where there is no movement blurring; and an actual world 
estimating step for estimating the pixel values of each of 
the pixels where there is no movement blurring, by computing 
the normal equations generated from the normal equation 
generating step. 

With the present invention, a light signal of the real 
world is projected on multiple pixels, each of which has a 
time integration effect, and assuming a processing region is 
set within the image data wherein a portion of the 
continuity of the light signal of the real world has been 
lost, and a movement vector of the object within the image 
data corresponding to the continuity of the light signal of 
the real world wherein a portion of the continuity of the 
image data is lost, is set, and the pixel values of each of 
the pixels within the processing region are said to be a 
value integrated while the pixel values of each of the 
pixels not generating movement blurring corresponding to the 
object is shifted corresponding to the movement vector, and 
the relation between the pixel values of each of the pixels 
of the processing region and the pixel values of each of the 
pixels not generating movement blurring is modeled, and a 
normal equation is generated as to the model, from the first 
equation wherein the pixel values of each of the pixels 
within the processing region are substituted and the second 
equation which constrains the relation between each of the 
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pixels not generating movement blurring, and by computing 
the generated normal equation, pixel values for each of the 
pixels not generating movement blurring can be estimated. 
Brief Description of the Drawings 
5 Fig. 1 is a diagram illustrating the principle of the 

present invention . 

Fig. 2 is a block diagram illustrating an example of 
the hardware configuration of a signal processing device 4. 

Fig. 3 is a block diagram illustrating a configuration 
10 example of an embodiment of the signal processing device 4 
shown in Fig . 1 . 

Fig. 4 is a diagram describing in detail the principle 
of the signal processing performed by the signal processing 
device 4 . 

15 Fig. 5 is a diagram for describing an example of array 

of pixels on an image sensor. 

Fig. 6 is a diagram for describing the operations of a 
detecting device which is a CCD. 

Fig. 7 is a diagram for describing the relation between 
20 light cast into detecting elements corresponding to pixel D 
through pixel F, and pixel values. 

Fig. 8 is a diagram describing the relation between 
elapsing of time, light cast into a detecting element 
corresponding to one pixel, and pixel values. 
25 Fig. 9 is a diagram illustrating an example of an image 
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of a linear object in the actual world 1. 

Fig. 10 is a diagram illustrating an example of pixel 
values of image data obtained by actual imaging. 

Fig. 11 is a diagram illustrating an example of an 
5 image of the actual world 1, of an object which is of a 
color different from that of the background, having a 
monotone and linear edge. 

Fig. 12 is a diagram illustrating an example of pixel 
values of image data obtained by actual imaging. 
10 Fig. 13 is a schematic diagram of image data. 

Fig. 14 is a diagram for describing estimation of a 
model 161 with M pieces of data 162. 

Fig. 15 is a diagram for describing the relation 
between signals of the actual world 1 and data 3. 
15 pig. 16 is a diagram illustrating an example of data 3 

of interest at the time of creating an Expression. 

Fig. 17 is a diagram for describing signals for two 
objects in the actual world 1, and values belonging to a 
mixed region at the time of creating an expression. 
20 Fig. 18 is a diagram for describing continuity 

represented by Expression (18), Expression (19), and 
Expression (22) . 

Fig. 19 is a diagram illustrating an example of M 
pieces of data 162 extracted from data 3. 
25 Fig. 20 is a diagram for describing integration of 
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signals of the actual world 1 in the time direction and two- 
dimensional spatial direction, in the data 3. 

Fig. 21 is a diagram for describing an integration 
region at the time of generating high-resolution data with 
5 higher resolution in the spatial direction. 

Fig. 22 is a diagram for describing an integration 
region at the time of generating high-resolution data with 
higher resolution in the time direction. 

Fig. 23 is a diagram for describing an integration 
10 region at the time of generating high-resolution data with 
higher resolution in the time-spatial directions. 

Fig. 24 is a diagram illustrating the original image of 
the input image . 

Fig. 25 is a diagram illustrating an example of an 

15 input image. 

Fig. 26 is a diagram illustrating an image obtained by 
applying conventional class classification adaptation 
processing . 

Fig. 27 is a diagram illustrating results of detecting 
20 a fine line region. 

Fig. 28 is a diagram illustrating an example of an 
output image output from the signal processing device 4. 

Fig. 29 is a flowchart for describing signal processing 
with the signal processing device 4. 
25 Fig. 30 is a block diagram illustrating the 
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configuration of a data continuity detecting unit 101. 

Fig. 31 is a diagram illustrating an image in the 
actual world 1 with a fine line in front of the background. 

Fig. 32 is a diagram for describing approximation of a 
background with a plane. 

Fig. 33 is a diagram illustrating the cross-sectional 
shape of image data regarding which the image of a fine line 
has been projected. 

Fig. 34 is a diagram illustrating the cross-sectional 
shape of image data regarding which the image of a fine line 
has been projected. 

Fig. 35 is a diagram illustrating the cross-sectional 
shape of image data regarding which the image of a fine line 
has been projected. 

Fig. 36 is a diagram for describing the processing for 
detecting a peak and detecting of monotonous 
increase/decrease regions . 

Fig. 37 is a diagram for describing the processing for 
detecting a fine line region wherein the pixel value of the 
peak exceeds a threshold, while the pixel value of the 
adjacent pixel is equal to or below the threshold value. 

Fig. 38 is a diagram representing the pixel value of 
pixels arrayed in the direction indicated by dotted line AA' 
in Fig. 37. 

Fig. 3 9 is a diagram for describing processing for 
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detecting continuity in a monotonous increase/decrease 
region. 

Fig. 40 is a diagram illustrating an example of other 
processing for detecting regions where an image of a fine 
line has been projected. 

Fig. 41 is a flowchart for describing continuity 
detection processing. 

Fig. 42 is a diagram for describing processing for 
detecting continuity of data in the time direction. 

Fig. 43 is a block diagram illustrating the 
configuration of a non-continuity component extracting unit 
201. 

Fig. 44 is a diagram for describing the number of time 
of rejections . 

Fig. 45 is a flowchart describing the processing for 
extracting the non-continuity component. 

Fig. 46 is a flowchart describing the processing for 
extracting the continuity component. 

Fig. 47 is a flowchart describing other processing for 
extracting the continuity component. 

Fig. 48 is a flowchart describing still other 
processing for extracting the continuity component. 

Fig. 49 is a block diagram illustrating another 
configuration of a~ continuity component extracting unit 101. 

Fig. 50 is a diagram for describing the activity in an 
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input image having data continuity. 

Fig. 51 is a diagram for describing a block for 
detecting activity. 

Fig. 52 is a diagram for describing the angle of data 
continuity as to activity. 

Fig. 53 is a block diagram illustrating a detailed 
configuration of the data continuity detecting unit 101. 

Fig. 54 is a diagram describing a set of pixels. 

Fig. 55 is a diagram describing the relation between 
the position of a pixel set and the angle of data continuity. 

Fig. 56 is a flowchart for describing processing for 
detecting data continuity. 

Fig. 57 is a diagram illustrating a set of pixels 
extracted when detecting the angle of data continuity in the 
time direction and space direction. 

Fig. 58 is a diagram for describing the principle of 
function approximation, which is an example of an embodiment 
of the actual world estimating unit shown in Fig. 3. 

Fig. 59 is a diagram for describing integration effects 
in the event that the sensor is a CCD. 

Fig. 60 is a diagram for describing a specific example 
of the integration effects of the sensor shown in Fig. 59. 

Fig. 61 is a diagram for describing another specific 
example of the integration effects of the sensor shown in 
Fig. 59. 
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Fig. 62 is a diagram representing a fine-line-inclusive 
actual world region shown in Fig. 60. 

Fig. 63 is a diagram for describing the principle of an 
example of an embodiment of the actual world estimating unit 
5 shown in Fig. 3, in comparison with the example shown in Fig. 
58 . 

Fig. 64 is a diagram representing the fine-line- 
inclusive data region shown in Fig. 60. 

Fig. 65 is a diagram wherein each of the pixel values 
10 contained in the fine-line-inclusive data region shown in 
Fig. 64 are plotted on a graph. 

Fig. 66 is a diagram wherein an approximation function, 
approximating the pixel values contained in the fine -line- 
inclusive data region shown in Fig. 65, is plotted on a 
15 graph. 

Fig. 67 is a diagram for describing the continuity in 
the spatial direction which the fine-line-inclusive actual 
world region shown in Fig. 60 has. 

Fig. 68 is a diagram wherein each of the pixel values 
20 contained in the fine-line-inclusive data region shown in 
Fig. 64 are plotted on a graph. 

Fig. 69 is a diagram for describing a state wherein 
each of the input pixel values indicated in Fig. 68 are 
shifted by a predetermined shift amount. 
25 Fig. 70 is a diagram wherein an approximation function, 
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approximating the pixel values contained in the fine-line- 
inclusive data region shown in Fig. 65, is plotted on a 
graph, taking into consideration the spatial-direction 
continuity. 

Fig. 71 is a diagram for describing space-mixed region. 

Fig. 72 is a diagram for describing an approximation 
function approximating actual -world signals in a space-mixed 
region. 

Fig. 73 is a diagram wherein an approximation function, 
approximating the actual world signals corresponding to the 
fine-line-inclusive data region shown in Fig. 65, is plotted 
on a graph, taking into consideration both the sensor 
integration properties and the spatial -direction continuity. 

Fig. 74 is a block diagram for describing a 
configuration example of the actual world estimating unit 
using, of function approximation techniques having the 
principle shown in Fig. 58, primary polynomial approximation. 

Fig. 75 is a flowchart for describing actual world 
estimation processing which the actual world estimating unit 
of the configuration shown in Fig. 74 executes. 

Fig. 76 is a diagram for describing a tap range. 

Fig. 77 is a diagram for describing actual world 
signals having continuity in the spatial direction. 

Fig. 78 is a diagram for describing integration effects 
in the event that the sensor is a CCD. 
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Fig. 79 is a diagram for describing distance in the 
cross -sectional direction . 

Fig. 80 is a block diagram for describing a 
configuration example of the actual world estimating unit 
5 using, of function approximation techniques having the 
principle shown in Fig. 58, quadratic polynomial 
approximat ion . 

Fig. 81 is a flowchart for describing actual world 
estimation processing which the actual world estimating unit 
10 of the configuration shown in Fig. 80 executes. 

Fig. 82 is a diagram for describing a tap range. 
Fig. 83 is a diagram for describing direction of 
continuity in the time-spatial direction. 

Fig. 84 is a diagram for describing integration effects' 
15 in the event that the sensor is a CCD. 

Fig. 85 is a diagram for describing actual world 
signals having continuity in the spatial direction. 

Fig. 86 is a diagram for describing actual world 
signals having continuity in the space-time directions. 
20 Fig. 87 is a block diagram for describing a 

configuration example of the actual world estimating unit 
using, of function approximation techniques having the 
principle shown in Fig. 58, cubic polynomial approximation. 
Fig. 8 8 is a flowchart for describing actual world 
25 estimation processing which the actual world estimating unit 
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of the configuration shown in Fig. 87 executes. 

Fig. 89 is a diagram for describing the principle of 
re- integration, which is an example of an embodiment of the 
image generating unit shown in Fig. 3. 
5 Fig. 90 is a diagram for describing an example of an 

input pixel and an approximation function for approximation 
of an actual world signal corresponding to the input pixel. 

Fig. 91 is a diagram for describing an example of 
creating four high-resolution pixels in the one input pixel 
10 shown in Fig. 90, from the approximation function shown in 
Fig. 90. 

Fig. 92 is a block diagram for describing a 
configuration example of an image generating unit using, of 
re-integration techniques having the principle shown in Fig. 
15 89, one -dimensional re- integration technique. 

Fig. 93 is a flowchart for describing the image 
generating processing which the image generating unit of the 
configuration shown in Fig. 92 executes. 

Fig. 94 is a diagram illustrating an example of the 
2 0 original image of the input image. 

Fig. 95 is a diagram illustrating an example of image 
data corresponding to the image shown in Fig. 94. 

Fig. 96 is a diagram representing an example of an 
input image . 

25 Fig. 97 is a diagram representing an example of image 
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data corresponding to the image shown in Fig. 96. 

Fig. 98 is a diagram illustrating an example of an 
image obtained by subjecting an input image to conventional 
class classification adaptation processing. 
5 Fig. 99 is a diagram representing an example of image 

data corresponding to the image shown in Fig. 98. 

Fig. 100 is a diagram illustrating an example of an 
image obtained by subjecting an input image to the one- 
dimensional re- integration technique. 
10 Fig. 101 is a diagram illustrating an example of image 

data corresponding to the image shown in Fig. 10 0. 

Fig. 102 is a diagram for describing actual -world 
signals having continuity in the spatial direction. 
Fig. 103 is a block diagram for describing a 
15 configuration example of an image generating unit which uses, 
of the re-integration techniques having the principle shown 
in Fig. 89, a two-dimensional re- integration technique. 

Fig. 104 is a diagram for describing distance in the 
cross -sectional direction . 
20 Fig. 105 is a flowchart for describing the image 

generating processing which the image generating unit of the 
configuration shown in Fig. 103 executes. 

Fig. 106 is a diagram for describing an example of an 
input pixel . 

25 Fig. 107 is a diagram for describing an example of 



- 19 - 

S04P0850 

creating four high-resolution pixels in the one input pixel 
shown in Fig. 106, with the two-dimensional re- integration 
technique . 

Fig. 108 is a diagram for describing the direction of 
5 continuity in the space-time directions. 

Fig. 109 is a block diagram for describing a 
configuration example of the image generating unit which 
uses, of the re -integration techniques having the principle 
shown in Fig. 89, a three-dimensional re-integration 
10 technique. 

Fig. 110 is a flowchart for describing the image 
generating processing which the image generating unit of the 
configuration shown in Fig. 109 executes. 

Fig. Ill is a block diagram illustrating a 
15 configuration example of another embodiment of the signal 
processing device 4 shown in Fig. 1. 

Fig. 112 is a flowchart for describing the processing 
of the signal processing device 4 shown in Fig. 111. 
Fig. 113 is a block diagram illustrating a 
20 configuration example of an applied embodiment of the signal 
processing device 4 shown in Fig. 111. 

Fig. 114 is a flowchart for describing the processing 
of the signal processing device 4 shown in Fig. 113. 

Fig. 115 is a diagram describing a light signal of an 
25 actual world 1. 
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Fig. 116 is a diagram describing the integration effect 
in the event that the sensor 2 is a CCD. 

Fig. 117 is a diagram showing an approximation function 
f(x,y) of the actual world 1. 
5 Fig. 118 is a diagram describing an input image to be 

input in the signal processing device 4 of Fig. 113, and a 
processing region. 

Fig. 119 is a diagram describing an example of a 
processing region set in the signal processing device 4 of 
10 the Fig. 113 . 

Fig. 12 0 is a diagram illustrating an approximation 
function f (x) of the X cross- sect ion, wherein the value of y 
within the approximation function f (x,y) in Fig. 120 is a 
predetermined value y c (y = y c ) • 
15 Fig. 121 is a diagram showing the approximation 

function f (x) in Fig. 120 after l/v time has passed. 

Fig. 122 is a diagram showing the approximation 
function f (x) in Fig. 121 after l/v time has passed. 

Fig. 123 is a diagram describing the movement of the 
2 0 object to be imaged in the input image moves v pixels for 
every shutter time. 

Fig. 124 is a diagram showing the approximation 
function f (x) of y = y c at a predetermined precise timing, 
illustrated in Fig. 120, with a X - t plane. 
2 5 Fig. 12 5 is a diagram showing pixel values P 0 through P 9 
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generating movement blurring, with a X - t plane, so as to 
correspond with the pixel values Q 0 through Q 9 in Fig. 124. 

Fig. 126 is a diagram showing the pixel values P 0 
through P 9 of an input image with the pixel values Q 0 through 
5 Q 9 in Fig . 124 . 

Fig. 12 7 is a diagram describing a method for solving 
the blank regions in Fig. 126, hypothesizing the edge 
portions of the processing region as "being flat". 

Fig. 128 is a diagram describing the line of interest 
10 of the processing region in Fig. 119. 

Fig. 129 is a block diagram illustrating a 
configuration example of the actual world estimating unit 
15013 in Fig. 113 . 

Fig. 130 is a flowchart describing the actual world 
15 estimating processing of step S15008 in Fig. 114. 

Fig. 131 is a flowchart describing the actual world 
estimating processing of step S15008 in Fig. 114. 

Fig. 132 is a diagram illustrating the input image to 
be input into the signal processing device 4 in Fig. 113. 
20 Fig. 133 is a diagram illustrating a processed and 

output image wherein the signal processing device 4 in Fig. 
113 has processed the input image in Fig. 132 with the first 
method, which adds an equation which is postulated as having 
the edge portions of the processing regions as "being flat". 
25 Fig. 134 is a diagram illustrating a processed and 
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output image wherein the signal processing device 4 in Fig. 

113 has processed the input image in Fig. 132 with the 

second method, which adds a conditional expression 

constraining the relation between the adjacent pixel values. 
5 Fig. 135 is a block diagram of a configuration example 

of another embodiment of an application example of the 

signal processing device 4 in Fig. 111. 

Fig. 136 is a flowchart describing the processing of 

the signal processing device 4 in Fig. 135. 
10 Fig. 137 is a block diagram illustrating a 

configuration example of the actual world estimating unit 

15083 in Fig. 135 . 

Fig. 138 is a flowchart describing the actual world 

estimating processing in step S15088 in Fig. 136. 
15 Fig. 13 9 is a flowchart describing the actual world 

estimating processing in step S15088 in Fig. 136. 

Fig. 14 0 is a block diagram of a configuration example 

of another embodiment of an application example of the 

signal processing device 4 in Fig. 111. 
2 0 Fig. 141 is a flowchart describing the processing of 

the signal processing device 4 in Fig. 140. 

Fig. 142 is a block diagram illustrating a 

configuration example of the actual world estimating unit 

15113 in Fig. 140. 
25 Fig. 143 is a flowchart describing the actual world 
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estimating processing in step S15168 in Fig. 141. 

Fig. 144 is a diagram illustrating the input image 
which is input into the signal processing device 4 in Fig. 
140 . 

5 Fig. 145 is a diagram illustrating an example of an 

output image wherein the signal processing device 4 in Fig. 
135 processes the input image in Fig. 144 using the 
weighting W bj corresponding to the constraint condition 
expression as the same weighting for all of the constraint 

10 condition expressions. 

Fig. 146 is a diagram illustrating an example of an 
output image wherein the signal processing device 4 in Fig. 
140 processes the input image in Fig. 144 using the 
weighting W b j corresponding to the constraint condition 

15 expression as a 0 or a 1 , according to the activity of the 
input image . 

Fig. 14 7 is a block diagram of a configuration example 
of another embodiment of an application example of the 
signal processing device 4 in Fig. 111. 
20 Fig. 148 is a flowchart describing the processing of 

the signal processing device 4 in Fig. 147. 

Fig. 149 is a block diagram illustrating a 
configuration example of the actual world estimating unit 
15153 in Fig. 147 . 
25 Fig. 150 is a flowchart for describing the actual world 
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estimating processing in step S15218 in Fig. 148. 

Fig. 151 is a flowchart for describing the actual world 
estimating processing in step S15218 in Fig. 148. 
Fig. 152 is a block diagram illustrating a 
5 configuration example of a continuity setting unit 15012 
shown in Fig. 113. 

Fig. 153 is a diagram for describing amount of movement. 
Fig. 154 is a diagram illustrating pixel values of an 
image output from a camera, taken by the camera while a 
10 foreground object passes in front of a background object. 

Fig. 155 is a diagram illustrating difference values of 
the pixel values of pixels in the image shown in Fig. 154. 

Fig. 156 is a flowchart for describing processing for 
detecting amount of movement. 
15 Fig. 157 is a flowchart for describing processing for 

detecting correlation. 

Best Mode for Carrying Out the Invention 
Fig. 1 illustrates the principle of the present 
invention. As shown in the drawing, events (phenomena) in 
2 0 an actual world 1 having dimensions of space, time, and mass, 
are acquired by a sensor 2, and formed into data. Events in 
the actual world 1 refer to light (images) , sound, pressure, 
temperature, mass, humidity, brightness/darkness, or smells, 
and so forth. The events in the actual world 1 are 
25 distributed in the space-time directions. For example, an 
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image of the actual world 1 is a distribution of the 
intensity of light of the actual world 1 in the space-time 
directions . 

Taking note of the sensor 2, of the events in the 
5 actual world 1 having the dimensions of space, time, and 
mass, the events in the actual world 1 which the sensor 2 
can acquire, are converted into data 3 by the sensor 2. It 
can be said that information indicating events in the actual 
world 1 are acquired by the sensor 2 . 
10 That is to say, the sensor 2 converts information 

indicating events in the actual world 1, into data 3. It 
can be said that signals which are information indicating 
the events (phenomena) in the actual world 1 having 
dimensions of space, time, and mass, are acquired by the 
15 sensor 2 and formed into data. 

Hereafter, the distribution of events such as images, 
sound, pressure, temperature, mass, humidity, 

brightness/darkness, or smells, and so forth, in the actual 
world 1, will be referred to as signals of the actual world 

20 1, which are information indicating events. Also, signals 
which are information indicating events of the actual world 
1 will also be referred to simply as signals of the actual 
world 1. In the present Specification, signals are to be 
understood to include phenomena and events, and also include 

25 those wherein there is no intent on the transmitting side. 
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The data 3 (detected signals) output from the sensor 2 
is information obtained by projecting the information 
indicating the events of the actual world 1 on a space-time 
having a lower dimension than the actual world 1 . For 
5 example, the data 3 which is image data of a moving image, 

is information obtained by projecting an image of the three- 
dimensional space direction and time direction of the actual 
world 1 on the time-space having the two-dimensional space 
direction and time direction. Also, in the event that the 

10 data 3 is digital data for example, the data 3 is rounded 

off according to the sampling increments. In the event that 
the data 3 is analog data, information of the data 3 is 
either compressed according to the dynamic range, or a part 
of the information has been deleted by a limiter or the like. 

15 Thus, by projecting the signals shown are information 

indicating events in the actual world 1 having a 
predetermined number of dimensions onto data 3 (detection 
signals) , a part of the information indicating events in the 
actual world 1 is lost. That is to say, a part of the 

20 information indicating events in the actual world 1 is lost 
from the data 3 which the sensor 2 outputs. 

However, even though a part of the information 
indicating events in the actual world 1 is lost due to 
projection, the data 3 includes useful information for 

25 estimating the signals which are information indicating 
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events (phenomena) in the actual world 1. 

With the present invention, information having 
continuity contained in the actual world 1 or the data 3 is 
used as useful information for estimating the signals which 
5 is information of the actual world 1. Continuity is a 
concept which is newly defined. 

Taking note of the actual world 1, events in the actual 
world 1 include characteristics which are constant in 
predetermined dimensional directions. For example, an 

10 object (corporeal object) in the actual world 1 either has 
shape, pattern, or color that is continuous in the space 
direction or time direction, or has repeated patterns of 
shape, pattern, or color. 

Accordingly, the information indicating the events in 

15 actual world 1 includes characteristics constant in a 
predetermined dimensional direction. 

With a more specific example, a linear object such as a 
string, cord, or rope, has a characteristic which is 
constant in the length-wise direction, i.e., the spatial 

20 direction, that the cross-sectional shape is the same at 
arbitrary positions in the length-wise direction. The 
continuity in the spatial direction that the cross-sectional 
shape is the same at arbitrary positions in the length-wise 
direction comes from the characteristic that the linear 

25 object is long. 
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Accordingly, an image of the linear object has a 
characteristic which is constant in the length-wise 
direction, i.e., the spatial direction, that the cross- 
sectional shape is the same, at arbitrary positions in the 
5 length-wise direction. 

Also, a monotone object, which is a corporeal object, 
having an expanse in the spatial direction, can be said to 
have a continuity of having the same color in the spatial 
direction regardless of the part thereof. 

10 In the same way, an image of a monotone object, which 

is a corporeal object, having an expanse in the spatial 
direction, have a continuity of having the same color in the 
spatial direction regardless of the part thereof. 

In this way, events in the actual world 1 (real world) 

15 have characteristics which are constant in predetermined 
dimensional directions, so signals of the actual world 1 
have characteristics which are constant in predetermined 
dimensional directions 

In the present Specification, such characteristics 

2 0 which are constant in predetermined dimensional directions 

will be called continuity. Continuity of the signals of the 
actual world 1 (real world) means the characteristics which 
are constant in predetermined dimensional directions which 
the signals indicating the events of the actual world 1 

25 (real world) have. 
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Countless such continuities exist in the actual world 1 
(real world) . 

Next, taking note of the data 3, the data 3 is obtained 
by signals which is information indicating events of the 
5 actual world 1 having predetermined dimensions being 
projected by the sensor 2, and includes continuity 
corresponding to the continuity of signals in the real world. 
It can be said that the data 3 includes continuity wherein 
the continuity of actual world signals has been projected. 
10 However, as described above, in the data 3 output from 

the sensor 2, a part of the information of the actual world 
1 has been lost, so a part of the continuity contained in 
the signals of the actual world 1 (real world) may be lost 
from the data. 

15 In other words, the data 3 contains at least a part of 

the continuity within the continuity of the signals of the 
actual world 1 (real world) as data continuity. Data 
continuity means characteristics which are constant in 
predetermined dimensional directions, which the data 3 has. 

20 With the present invention, continuity of the actual 

world 1 signals, or the data continuity which the data 3 has, 
is used as significant data for estimating signals which are 
information indicating events of the actual world 1. 

For example, with the signal processing device 4, 

25 information indicating an event in the actual world 1 which 
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has been lost is generated by signals processing of the data 

3, using data continuity. 

Now, with the signal processing device 4, of the length 
(space) , time, and mass, which are dimensions of signals 
5 serving as information indicating events in the actual world 
1, continuity in the spatial direction or time direction, 
are used. 

In Fig. 1, the sensor 2 is formed of, for example, a 
digital still camera, a video camera, or the like, and takes 
10 images of the actual world 1, and outputs the image data 

which is the obtained data 3, to a signal processing device 

4. The sensor 2 may also be a thermography device, a 
pressure sensor using photo-elasticity, or the like. 

The signal processing device 4 is configured of, for 
15 example, a personal computer or the like, and performs 
signal processing with regard to the data 3. 

The signal processing device 4 is configured as shown 
in Fig. 2, for example. A CPU (Central Processing Unit) 21 
executes various types of processing following programs 
20 stored in ROM (Read Only Memory) 22 or the storage unit 28. 

RAM (Random Access Memory) 23 stores programs to be executed 
by the CPU 21, data, and so forth, as suitable. The CPU 21, 
ROM 22, and RAM 23, are mutually connected by a bus 24. 
Also connected to the CPU 21 is an input/output 
25 interface 25 via the bus 24. An input device 26 made up of 
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a keyboard, mouse, microphone, and so forth, and an output 
unit 27 made up of a display, speaker, and so forth, are 
connected to the input/output interface 25. The CPU 21 
executes various types of processing corresponding to 
5 commands input from the input unit 26. The CPU 21 then 

outputs images and audio and the like obtained as a result 
of processing to the output unit 27. 

A storage unit 2 8 connected to the input /output 
interface 25 is configured of a hard disk for example, and 

10 stores the programs and various types of data which the CPU 
21 executes. A communication unit 2 9 communicates with 
external devices via the Internet and other networks. In 
the case of this example, the communication unit 29 acts as 
an acquiring unit for capturing data 3 output from the 

15 sensor 2 . 

Also, an arrangement may be made wherein programs are 
obtained via the communication unit 2 9 and stored in the 
storage unit 28. 

A drive 30 connected to the input/output interface 25 

20 drives a magnetic disk 51, optical disk 52, magneto -optical 
disk 53, or semiconductor memory 54 or the like mounted 
thereto, and obtains programs and data recorded therein. 
The obtained programs and data are transferred to the 
storage unit 28 as necessary and stored. 

25 Fig. 3 is a block diagram illustrating a signal 
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processing device 4 . 

Note that whether the functions of the signal 
processing device 4 are realized by hardware or realized by 
software is irrelevant. That is to say, the block diagrams 
5 in the present Specification may be taken to be hardware 

block diagrams or may be taken to be software function block 
diagrams . 

With the signal processing device 4 of which the 
configuration is shown in Fig. 3, image data which is an 

10 example of the data 3 is input, and the continuity of the 
data is detected from the input image data (input image) . 
Next, the signals of the actual world 1 acquired by the 
sensor 2 are estimated from the continuity of the data 
detected. Then, based on the estimated signals of the 

15 actual world 1, an image is generated, and the generated 

image (output image) is output. That is to say, Fig. 3 is a 
diagram illustrating the configuration of the signal 
processing device 4 which is an image processing device. 

The input image (image data which is an example of the 

2 0 data 3) input to the signal processing device 4 is supplied 
to a data continuity detecting unit 101 and actual world 
estimating unit 102. 

The data continuity detecting unit 101 detects the 
continuity of the data from the input image, and supplies 

2 5 data continuity information indicating the detected 
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continuity to the actual world estimating unit 102 and an 
image generating unit 103. The data continuity information 
includes, for example, the position of a region of pixels 
having continuity of data, the direction of a region of 
5 pixels having continuity of data (the angle or gradient of 
the time direction and space direction) , or the length of a 
region of pixels having continuity of data, or the like in 
the input image. Detailed configuration of the data 
continuity detecting unit 101 will be described later. 

10 The actual world estimating unit 102 estimates the signals 

of the actual world 1, based on the input image and the data 
continuity information supplied from the data continuity 
detecting unit 101. That is to say, the actual world 
estimating unit 102 estimates an image which is the signals 

15 of the actual world cast into the sensor 2 at the time that 
the input image was acquired. The actual world estimating 
unit 102 supplies the actual world estimation information 
indicating the results of the estimation of the signals of 
the actual world 1, to the image generating unit 103. The 

20 detailed configuration of the actual world estimating unit 
102 will be described later. 

The image generating unit 103 generates signals further 
approximating the signals of the actual world 1, based on 
the actual world estimation information indicating the 

25 estimated signals of the actual world 1, supplied from the 
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actual world estimating unit 102, and outputs the generated 
signals. Or, the image generating unit 103 generates 
signals further approximating the signals of the actual 
world 1, based on the data continuity information supplied 
5 from the data continuity detecting unit 101, and the actual 
world estimation information indicating the estimated 
signals of the actual world 1, supplied from the actual 
world estimating unit 102, and outputs the generated signals. 
That is to say, the image generating unit 103 generates 

10 an image further approximating the image of the actual world 
1 based on the actual world estimation information, and 
outputs the generated image as an output image. Or, the 
image generating unit 103 generates an image further 
approximating the image of the actual world 1 based on the 

15 data continuity information and actual world estimation 

information, and outputs the generated image as an output 
image . 

For example, the image generating unit 103 generates an 
image with higher resolution in the spatial direction or 

2 0 time direction in comparison with the input image, by 

integrating the estimated image of the actual world 1 within 
a desired range of the spatial direction or time direction, 
based on the actual world estimation information, and 
outputs the generated image as an output image. For example, 

25 the image generating unit 103 generates an image by 
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extrapolation/interpolation, and outputs the generated image 
as an output image . 

Detailed configuration of the image generating unit 103 
will be described later. 
5 Next, the principle of the present invention will be 

described with reference to Fig. 4. 

For example, signals of the actual world 1, which are 
an image for example, are imaged on the photoreception face 
of a CCD (Charge Coupled Device) which is an example of the 

10 sensor 2. The CCD, which is an example of the sensor 2, has 
integration properties, so difference is generated in the 
data 3 output from the CCD as to the image of the actual 
world 1. Details of the integration properties of the 
sensor 2 will be described later. 

15 With the signal processing by the signal processing 

device 4, the relation between the image of the actual world 
1 obtained by the CCD, and the data 3 taken by the CCD and 
output, is explicitly taken into consideration. That is to 
say, the relation between the data 3 and the signals which 

20 is information of the actual world obtained by the sensor 2, 
is explicitly taken into consideration. 

More specifically, as shown in Fig. 4, the signal 
processing device 4 uses a model 161 to approximate 
(describe) the actual world 1. The model 161 is represented 

25 by, for example, N variables. More accurately, the model 
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161 approximates (describes) signals of the actual world 1. 

In order to predict the model 161, the signal 
processing device 4 extracts M pieces of data 162 from the 
data 3 . At the time of extracting the M pieces of data 162 
5 from the data 3, the signal processing device 4 uses the 

continuity of the data contained in the data 3, for example. 
In other words, the signal processing device 4 extracts data 

162 for predicting the model 161, based on the continuity of 
the data contained in the data 3. Consequently, in this 

10 case, the model 161 is constrained by the continuity of the 
data . 

That is to say, the model 161 approximates (information 
(signals) indicating) events of the actual world 1 having 
continuity (continuity in a predetermined dimensional 
15 direction) , which generates the data continuity in the data 
3 when acquired with the sensor 2 . 

Now, in the event that the number M of the data 162 is 
N or more, which is the number of variables of the model, 
the model 161 represented by the N variables can be 
20 predicted, from the M pieces of the data 162. 

In this way, the signal processing device 4 can take 
into consideration the signals which are information of the 
actual world 1, by predicting the model 161 approximating 
(describing) the (signals of the) actual world 1. 
25 Next, the integration effects of the sensor 2 will be 



- 37 - 

S04P0850 

described. 

An image sensor such as a CCD or CMOS (Complementary 
Metal -Oxide Semiconductor) sensor or the like, which is the 
sensor 2 for taking images, projects signals, which are 
5 information of the real world, onto two-dimensional data, at 
the time of imaging the real world. The pixels of the image 
sensor each have a predetermined area, as a so-called 
photoreception face (photoreception region) . Incident light 
to the photoreception face having a predetermined area is 
10 integrated in the space direction and time direction for 

each pixel, and is converted into a single pixel value for 
each pixel . 

The space-time integration of images will be described 
with reference to Fig. 5 through Fig. 8. 

15 An image sensor images a subject (object) in the real 

world, and outputs the obtained image data as a result of 
imagining in increments of single frames. That is to say, 
the image sensor acquires signals of the actual world 1 
which is light reflected off of the subject of the actual 

20 world 1, and outputs the data 3. 

For example, the image sensor outputs image data of 3 0 
frames per second. In this case, the exposure time of the 
image sensor can be made to be 1/3 0 seconds. The exposure 
time is the time from the image sensor starting conversion 

25 of incident light into electric charge, to ending of the 
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conversion of incident light into electric charge. 
Hereafter, the exposure time will also be called shutter 
time . 

Fig. 5 is a diagram describing an example of a pixel 
5 array on the image sensor. In Fig. 5, A through I denote 
individual pixels. The pixels are placed on a plane 
corresponding to the image displayed by the image data. A 
single detecting element corresponding to a single pixel is 
placed on the image sensor. At the time of the image sensor 

10 taking images of the actual world 1, the one detecting 

element outputs one pixel value corresponding to the one 
pixel making up the image data. For example, the position 
in the spatial direction X (X coordinate) of the detecting 
element corresponds to the horizontal position on the image 

15 displayed by the image data, and the position in the spatial 
direction Y (Y coordinate) of the detecting element 
corresponds to the vertical position on the image displayed 
by the image data. 

Distribution of intensity of light of the actual world 

20 1 has expanse in the three-dimensional spatial directions 

and the time direction, but the image sensor acquires light 
of the actual world 1 in two-dimensional spatial directions 
and the time direction, and generates data 3 representing 
the distribution of intensity of light in the two- 

25 dimensional spatial directions and the time direction. 
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As shown in Fig. 6, the detecting device which is a CCD 
for example, converts light cast onto the photoreception 
face (photoreception region) (detecting region) into 
electric charge during a period corresponding to the shutter 
5 time, and accumulates the converted charge. The light is 
information (signals) of the actual world 1 regarding which 
the intensity is determined by the three-dimensional spatial 
position and point - in- time . The distribution of intensity 
of light of the actual world 1 can be represented by a 

10 function F(x, y, z, t) , wherein position x, y, z, in three- 
dimensional space, and point-in- time t, are variables. 

The amount of charge accumulated in the detecting 
device which is a CCD is approximately proportionate to the 
intensity of the light cast onto the entire photoreception 

15 face having two-dimensional spatial expanse, and the amount 
of time that light is cast thereupon. The detecting device 
adds the charge converted from the light cast onto the 
entire photoreception face, to the charge already 
accumulated during a period corresponding to the shutter 

20 time. That is to say, the detecting device integrates the 

light cast onto the entire photoreception face having a two- 
dimensional spatial expanse, and accumulates a change of an 
amount corresponding to the integrated light during a period 
corresponding to the shutter time. The detecting device can 

25 also be said to have an integration effect regarding space 
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(photoreception face) and time (shutter time) . 

The charge accumulated in the detecting device is 
converted into a voltage value by an unshown circuit, the 
voltage value is further converted into a pixel value such 
5 as digital data or the like, and is output as data 3. 

Accordingly, the individual pixel values output from the 
image sensor have a value projected on one -dimensional space, 
which is the result of integrating the portion of the 
information (signals) of the actual world 1 having time- 

10 space expanse with regard to the time direction of the 

shutter time and the spatial direction of the photoreception 
face of the detecting device. 

That is to say, the pixel value of one pixel is 
represented as the integration of F(x, y, t) . F(x, y, t) is 

15 a function representing the distribution of light intensity 
on the photoreception face of the detecting device. For 
example, the pixel value P is represented by Expression (1) . 



2 0 at the left -side boundary of the photoreception face of the 
detecting device (X coordinate) . x 2 represents the spatial 
coordinate at the right-side boundary of the photoreception 
face of the detecting device (X coordinate) . In Expression 




. . . (1) 



In Expression (1) , x represents the spatial coordinate 
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(1) , y represents the spatial coordinate at the top- side 
boundary of the photoreception face of the detecting device 
(Y coordinate) . y 2 represents the spatial coordinate at the 
bottom-side boundary of the photoreception face of the 
5 detecting device (Y coordinate) . Also, ti represents the 

point-in- time at which conversion of incident light into an 
electric charge was started. t 2 represents the point-in- 
time at which conversion of incident light into an electric 
charge was ended. 

10 Note that actually, the gain of the pixel values of the 

image data output from the image sensor is corrected for the 
overall frame, for example. 

Each of the pixel values of the image data are 
integration values of the light cast on the photoreception 

15 face of each of the detecting elements of the image sensor, 
and of the light cast onto the image sensor, waveforms of 
light of the actual world 1 finer than the photoreception 
face of the detecting element are hidden in the pixel value 
as integrated values. 

20 Hereafter, in the present Specification, the waveform 

of signals represented with a predetermined dimension as a 
reference may be referred to simply as waveforms. 

Thus, the image of the actual world 1 (light signals) 
is integrated in the spatial direction and time direction in 

25 increments of pixels, so a part of the continuity of the 
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image of the actual world 1 drops out from the image data, 
so another part of the continuity of the image of the actual 
world 1 is left in the image data. Or, there may be cases 
wherein continuity which has changed from the continuity of 
5 the image of the actual world 1 is included in the image 
data . 

Further description will be made regarding the 
integration effect in the spatial direction for an image 
taken by an image sensor having integration effects. 

10 Fig. 7 is a diagram describing the relation between 

incident light to the detecting elements corresponding to 
the pixel D through pixel F, and the pixel values. F (x) in 
Fig. 7 is an example of a function representing the 
distribution of light intensity of the actual world 1, 

15 having the coordinate x in the spatial direction X in space 
(on the detecting device) as a variable. In other words, 
F (x) is an example of a function representing the 
distribution of light intensity of the actual world 1, with 
the spatial direction Y and time direction constant. In Fig. 

20 7, L indicates the length in the spatial direction X of the 
photoreception face of the detecting device corresponding to 
the pixel D through pixel F. 

The pixel value of a single pixel is represented as the 
integral of F (x) . For example, the pixel value P of the 

25 pixel E is represented by Expression (2) . 



p = J x 2 F(x)dx 
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In Expression (2) , x i represents the spatial coordinate 
in the spatial direction X at the left-side boundary of the 
photoreception face of the detecting device corresponding to 
the pixel E. x 2 represents the spatial coordinate in the 
spatial direction X at the right-side boundary of the 
photoreception face of the detecting device corresponding to 
the pixel E. 

In the same way, further description will be made 
regarding the integration effect in the time direction for 
an image taken by an image sensor having integration effects. 

Fig. 8 is a diagram for describing the relation between 
time elapsed, the incident light to a detecting element 
corresponding to a single pixel, and the pixel value. F(t) 
in Fig. 8 is a function representing the distribution of 
light intensity of the actual world 1, having the point -in- 
time t as a variable. In other words, F(t) is an example of 
a function representing the distribution of light intensity 
of the actual world 1, with the spatial direction Y and the 
spatial direction X constant. T s represents the shutter 
time . 

The frame #n-l is a frame which is previous to the 
frame #n time -wise, and the frame #n+l is a frame following 
the frame #n time-wise. That is to say, the frame #n-l, 
frame #n, and frame #n+l, are displayed in the order of 
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frame #n-l, frame #n, and frame #n+l. 

Note that in the example shown in Fig. 8, the shutter 
time t s and the frame intervals are the same. 

The pixel value of a single pixel is represented as the 
integral of F(t) . For example, the pixel value P of the 
pixel of frame #n is represented by Expression (3) . 



In Expression (3) , ti represents the time at which 
conversion of incident light into an electric charge was 
started. t 2 represents the time at which conversion of 
incident light into an electric charge was ended. 

Hereafter, the integration effect in the spatial 
direction by the sensor 2 will be referred to simply as 
spatial integration effect, and the integration effect in 
the time direction by the sensor 2 also will be referred to 
simply as time integration effect. Also, space integration 
effects or time integration effects will be simply called 
integration effects . 

Next, description will be made regarding an example of 
continuity of data included in the data 3 acquired by the 
image sensor having integration effects. 

Fig. 9 is a diagram illustrating a linear object of the 
actual world 1 (e.g., a fine line), i.e., an example of 




. . . (3) 
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distribution of light intensity. In Fig. 9, the position to 
the upper side of the drawing indicates the intensity 
(level) of light, the position to the upper right side of 
the drawing indicates the position in the spatial direction 
5 X which is one direction of the spatial directions of the 
image, and the position to the right side of the drawing 
indicates the position in the spatial direction Y which is 
the other direction of the spatial directions of the image. 
The image of the linear object of the actual world 1 

10 includes predetermined continuity. That is to say, the 
image shown in Fig. 9 has continuity in that the cross- 
sectional shape (the change in level as to the change in 
position in the direction orthogonal to the length 
direction) , at any arbitrary position in the length 

15 direction. 

Fig. 10 is a diagram illustrating an example of pixel 
values of image data obtained by actual image- taking , 
corresponding to the image shown in Fig. 9. 

That is to say, Fig. 10 is a model diagram of the image 

2 0 data obtained by imaging, with the image sensor, an image of 
a linear object having a diameter shorter than the length L 
of the photoreception face of each pixel, and extending in a 
direction offset from the array of the pixels of the image 
sensor (the vertical or horizontal array of the pixels) . 

2 5 The image cast into the image sensor at the time that the 
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image data shown in Fig. 10 was acquired is an image of the 
linear object of the actual world 1 shown in Fig. 9. 

In Fig. 10, the position to the upper side of the 
drawing indicates the pixel value, the position to the upper 
5 right side of the drawing indicates the position in the 
spatial direction X which is one direction of the spatial 
directions of the image, and the position to the right side 
of the drawing indicates the position in the spatial 
direction Y which is the other direction of the spatial 

10 directions of the image. The direction indicating the pixel 
value in Fig. 10 corresponds to the direction of level in 
Fig. 9, and the spatial direction X and spatial direction Y 
in Fig. 10 also are the same as the directions in Fig. 9. 
In the event of taking an image of a linear object 

15 having a diameter shorter than the length L of the 

photoreception face of each pixel with the image sensor, the 
linear object is represented in the image data obtained as a 
result of the image-taking as multiple arc shapes (half- 
discs) having a predetermined length which are arrayed in a 

20 diagonally-offset fashion, in a model representation, for 

example. The arc shapes are of approximately the same shape. 
One arc shape is formed on one row of pixels vertically, or 
is formed on one row of pixels horizontally. For example, 
one arc shape shown in Fig. 10 is formed on one row of 

25 pixels vertically. 
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Thus, with the image data taken and obtained by the 
image sensor for example, the continuity in that the cross- 
sectional shape in the spatial direction Y at any arbitrary 
position in the length direction is the same, which the 
5 linear object image of the actual world 1 had, is lost. 

Also, it can be said that the continuity, which the linear 
object image of the actual world 1 had, has changed into 
continuity in that arc shapes of the same shape formed on 
one row of pixels vertically or formed on one row of pixels 

10 horizontally are arrayed at predetermined intervals. 

Fig. 11 is a diagram illustrating an image in the 
actual world 1 of an object having a straight edge, and is 
of a monotone color different from that of the background, 
i.e., an example of distribution of light intensity. In Fig. 

15 11, the position to the upper side of the drawing indicates 
the intensity (level) of light, the position to the upper 
right side of the drawing indicates the position in the 
spatial direction X which is one direction of the spatial 
directions of the image, and the position to the right side 

2 0 of the drawing indicates the position in the spatial 

direction Y which is the other direction of the spatial 
directions of the image. 

The image of the object of the actual world 1 which has 
a straight edge and is of a monotone color different from 

2 5 that of the background, includes predetermined continuity. 
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That is to say, the image shown in Fig. 11 has continuity in 
that the cross-sectional shape (the change in level as to 
the change in position in the direction orthogonal to the 
edge) is the same at any arbitrary position in the length 
5 direction of the edge. 

Fig. 12 is a diagram illustrating an example of pixel 
values of the image data obtained by actual image- taking, 
corresponding to the image shown in Fig. 11. As shown in 
Fig. 12, the image data is in a stepped shape, since the 
10 image data is made up of pixel values in increments of 
pixels . 

Fig. 13 is a model diagram illustrating the image data 
shown in Fig. 12. 

The model diagram shown in Fig. 13 is a model diagram 

15 of image data obtained by taking, with the image sensor, an 
image of the object of the actual world 1 which has a 
straight edge and is of a monotone color different from that 
of the background, and extending in a direction offset from 
the array of the pixels of the image sensor (the vertical or 

20 horizontal array of the pixels) . The image cast into the 
image sensor at the time that the image data shown in Fig. 
13 was acquired is an image of the object of the actual 
world 1 which has a straight edge and is of a monotone color 
different from that of the background, shown in Fig. 11. 

25 In Fig. 13, the position to the upper side of the 
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drawing indicates the pixel value, the position to the upper 
right side of the drawing indicates the position in the 
spatial direction X which is one direction of the spatial 
directions of the image, and the position to the right side 
of the drawing indicates the position in the spatial 
direction Y which is the other direction of the spatial 
directions of the image. The direction indicating the pixel 
value in Fig. 13 corresponds to the direction of level in 
Fig. 11, and the spatial direction X and spatial direction Y 
in Fig. 13 also are the same as the directions in Fig. 11. 

In the event of taking an image of an object of the 
actual world 1 which has a straight edge and is of a 
monotone color different from that of the background with an 
image sensor, the straight edge is represented in the image 
data obtained as a result of the image- taking as multiple 
pawl shapes having a predetermined length which are arrayed 
in a diagonally-offset fashion, in a model representation, 
for example. The pawl shapes are of approximately the same 
shape. One pawl shape is formed on one row of pixels 
vertically, or is formed on one row of pixels horizontally. 
For example, one pawl shape shown in Fig. 13 is formed on 
one row of pixels vertically. 

Thus, the continuity of image of the object of the 
actual world 1 which has a straight edge and is of a 
monotone color different from that of the background, in 
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that the cross- sectional shape is the same at any arbitrary 
position in the length direction of the edge, for example, 
is lost in the image data obtained by imaging with an image 
sensor. Also, it can be said that the continuity, which the 
5 image of the object of the actual world 1 which has a 

straight edge and is of a monotone color different from that 
of the background had, has changed into continuity in that 
pawl shapes of the same shape formed on one row of pixels 
vertically or formed on one row of pixels horizontally are 

10 arrayed at predetermined intervals. 

The data continuity detecting unit 101 detects such 
data continuity of the data 3 which is an input image, for 
example. For example, the data continuity detecting unit 
101 detects data continuity by detecting regions having a 

15 continuity in a predetermined dimensional direction. For 
example, the data continuity detecting unit 101 detects a 
region wherein the same arc shapes are arrayed at constant 
intervals, such as shown in Fig. 10. Also, for example, the 
data continuity detecting unit 101 detects a region wherein 

2 0 the same pawl shapes are arrayed at constant intervals, such 
as shown in Fig. 13. 

Also, the data continuity detecting unit 101 detects 
continuity of the data by detecting angle (gradient) in the 
spatial direction, indicating an array of the same shapes. 

2 5 Also, for example, the data continuity detecting unit 
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101 detects continuity of data by detecting angle (movement) 
in the space direction and time direction, indicating the 
array of the same shapes in the space direction and the time 
direction. 

5 Further, for example, the data continuity detecting 

unit 101 detects continuity in the data by detecting the 
length of the region having continuity in a predetermined 
dimensional direction . 

Hereafter, the portion of data 3 where the sensor 2 has 
10 projected the image of the object of the actual world 1 
which has a straight edge and is of a monotone color 
different from that of the background, will also be called a 
two-valued edge. 

Now, with conventional signal processing, desired high- 
15 resolution data, for example, is generated from the data 3. 

Conversely, with the signal processing by the signal 
processing device 4, the actual world 1 is estimated from 
the data 3, and the high-resolution data is generated based 
on the estimation results. That is to say, the actual world 
20 1 is estimated from the data 3, and the high-resolution data 
is generated based on the estimated actual world 1, taking 
into consideration the data 3 . 

In order to generate the high-resolution data from the 
actual world 1, there is the need to take into consideration 
25 the relation between the actual world 1 and the data 3. For 
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example, how the actual world 1 is projected on the data 3 
by the sensor 2 which is a CCD, is taken into consideration. 

The sensor 2 which is a CCD has integration properties 
as described above. That is to say, one unit of the data 3 
5 (e.g., pixel value) can be calculated by integrating a 

signal of the actual world 1 with a detection region (e.g., 
photoreception face) of a detection device (e.g., CCD) of 
the sensor 2 . 

Applying this to the high-resolution data, the high- 
10 resolution data can be obtained by applying processing, 

wherein a virtual high-resolution sensor projects signals of 
the actual world 1 to the data 3, to the estimated actual 
world 1 . 

In other words, if the signals of the actual world 1 
15 can be estimated from the data 3, one value contained in the 

I 

high-resolution data can be obtained by integrating signals 
of the actual world 1 for each detection region of the 
detecting elements of the virtual high-resolution sensor (in 
the time-space direction) . 

2 0 For example, in the event that the change in signals of 

the actual world 1 are smaller than the size of the 
detection region of the detecting elements of the sensor 2, 
the data 3 cannot expresses the small changes in the signals 
of the actual world 1. Accordingly, high-resolution data 

25 indicating small change of the signals of the actual world 1 
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can be obtained by integrating the signals of the actual 
world 1 estimated from the data 3 with each region (in the 
time-space direction) that is smaller in comparison with the 
change in signals of the actual world 1. 
5 That is to say, integrating the signals of the 

estimated actual world 1 with the detection region with 
regard to each detecting element of the virtual high- 
resolution sensor enables the high-resolution data to be 
obtained . 

10 With the signal processing device 4, the image 

generating unit 103 generates the high-resolution data by 
integrating the signals of the estimated actual world 1 in 
the time-space direction regions of the detecting elements 
of the virtual high-resolution sensor, for example. 

15 Next, in order to estimate the actual world 1 from the 

data 3, at the signal processing device 4, the relationship 
between the data 3 and the actual world 1, continuity, and a 
spatial or temporal mixture in the data 3 (space mixture or 
time mixture) , are used. 

2 0 Here, a mixture means a value in the data 3 wherein the 

signals of two objects in the actual world 1 are mixed to 
yield a single value. 

A space mixture means the mixture of the signals of two 
objects in the spatial direction due to the spatial 

25 integration effects of the sensor 2. Time mixture will be 
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described later. 

The actual world 1 itself is made up of countless 
events, and accordingly, in order to represent the actual 
world 1 itself with mathematical expressions, for example, 
5 there is the need to have an infinite number of variables. 

It is impossible to predict all events of the actual world 1 
from the data 3 . 

In the same way, it is impossible to predict all of the 
signals of the actual world 1 from the data 3. 

10 Accordingly, with the signal processing device 4, of 

the signals of the actual world 1, a portion which has 
continuity and which can be expressed by the function f (x, y, 
z, t) is taken note of, and the portion of the signals of 
the actual world 1 which can be represented by the function 

15 f (x, y, z, t) and has continuity is approximated with a 

model 161 represented by N variables. As shown in Fig. 14, 
the model 161 is predicted from the M pieces of data 162 in 
the data 3 . 

In order to enable the model 161 to be predicted from 
20 the M pieces of data 162, first, there is the need to 
represent the model 161 with N variables based on the 
continuity, and second, to generate an expression using the 
N variables which indicates the relation between the model 
161 represented by the N variables and the M pieces of data 
25 162 based on the integral properties of the sensor 2. Since 
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the model 161 is represented by the N variables, based on 
the continuity, it can be said that the expression using the 
N variables that indicates the relation between the model 
161 represented by the N variables and the M pieces of data 
5 162, describes the relation between the part of the signals 
of the actual world 1 having continuity, and the part of the 
data 3 having data continuity. 

In other words, the part of the signals of the actual 
world 1 having continuity, that is approximated by the model 

10 161 represented by the N variables, generates data 
continuity in the data 3 . 

The data continuity detecting unit 101 detects the part 
of the data 3 where data continuity has been generated by 
the part of the signals of the actual world 1 having 

15 continuity, and the characteristics of the part where data 
continuity has been generated. 

For example, as shown in Fig. 15, in an image of the 
object of the actual world 1 which has a straight edge and 
is of a monotone color different from that of the background, 

2 0 the edge at the position of interest indicated by A in Fig. 
15, has a gradient. The arrow B in Fig. 15 indicates the 
gradient of the edge. A predetermined edge gradient can be 
represented as an angle as to a reference axis or as a 
direction as to a reference position. For example, a 

2 5 predetermined edge gradient can be represented as the angle 
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between the coordinates axis of the spatial direction X and 
the edge. For example, the predetermined edge gradient can 
be represented as the direction indicated by the length of 
the spatial direction X and the length of the spatial 
5 direction Y. 

At the time that the image of the object of the actual 
world 1 which has a straight edge and is of a monotone color 
different from that of the background is obtained at the 
sensor 2 and the data 3 is output, pawl shapes corresponding 

10 to the edge are arrayed in the data 3 at the position 

corresponding to the position of interest (A) of the edge in 
the image of the actual world 1, which is indicated by A 1 in 
Fig. 15, and pawl shapes corresponding to the edge are 
arrayed in the direction corresponding to the gradient of 

15 the edge of the image in the actual world 1, in the 

direction of the gradient indicated by B' in Fig. 15. 

The model 161 represented with the N variables 
approximates such a portion of the signals of the actual 
world 1 generating data continuity in the data 3 . 

2 0 At the time of formulating an expression using the N 

variables indicating the relation between the model 161 
represented with the N variables and the M pieces of data 
162, the values of part where data continuity is generated 
in the data 3 are used. 

25 In this case, in the data 3 shown in Fig. 16, taking 
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note of the values where data continuity is generated and 
which belong to a mixed region, an expression is formulated 
with a value integrating the signals of the actual world 1 
as being equal to a value output by the detecting element of 
5 the sensor 2. For example, multiple expressions can be 

formulated regarding the multiple values in the data 3 where 
data continuity is generated. 

In Fig. 16, A denotes the position of interest of the 
edge, and A' denotes (the position of) the pixel 

10 corresponding to the position (A) of interest of the edge in 
the image of the actual world 1 . 

Now, a mixed region means a region of data in the data 
3 wherein the signals for two objects in the actual world 1 
are mixed and become one value. For example, a pixel value 

15 wherein, in the image of the object of the actual world 1 
which has a straight edge and is of a monotone color 
different from that of the background in the data 3, the 
image of the object having the straight edge and the image 
of the background are integrated, belongs to a mixed region. 

20 Fig. 17 is a diagram describing signals for two objects 

in the actual world 1 and values belonging to a mixed region, 
in a case of formulating an expression. 

Fig. 17 illustrates, to the left, signals of the actual 
world 1 corresponding to two objects in the actual world 1 

25 having a predetermined expansion in the spatial direction X 
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and the spatial direction Y, which are acquired at the 
detection region of a single detecting element of the sensor 
2. Fig. 17 illustrates, to the right, a pixel value P of a 
single pixel in the data 3 wherein the signals of the actual 
5 world 1 illustrated to the left in Fig. 17 have been 

projected by a single detecting element of the sensor 2. 
That is to say, illustrates a pixel value P of a single 
pixel in the data 3 wherein the signals of the actual world 

1 corresponding to two objects in the actual world 1 having 
10 a predetermined expansion in the spatial direction X and the 

spatial direction Y which are acquired by a single detecting 

element of the sensor 2, have been projected. 

L in Fig. 17 represents the level of the signal of the 

actual world 1 which is shown in white in Fig. 17, 
15 corresponding to one object in the actual world 1. R in Fig. 

17 represents the level of the signal of the actual world 1 

which is shown hatched in Fig. 17, corresponding to the 

other object in the actual world 1. 

Here, the mixture ratio a is the ratio of (the area of) 
20 the signals corresponding to the two objects cast into the 

detecting region of the one detecting element of the sensor 

2 having a predetermined expansion in the spatial direction 
X and the spatial direction Y. For example, the mixture 
ratio a represents the ratio of area of the level L signals 

25 cast into the detecting region of the one detecting element 
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of the sensor 2 having a predetermined expansion in the 
spatial direction X and the spatial direction Y, as to the 
area of the detecting region of a single detecting element 
of the sensor 2 . 

5 In this case, the relation between the level L, level R, 

and the pixel value P, can be represented by Expression (4) . 

a xL+f (1-a) xR = P 

... (4) 

Note that there may be cases wherein the level R may be 

10 taken as the pixel value of the pixel in the data 3 

positioned to the right side of the pixel of interest, and 
there may be cases wherein the level L may be taken as the 
pixel value of the pixel in the data 3 positioned to the 
left side of the pixel of interest. 

15 Also, the time direction can be taken into 

consideration in the same way as with the spatial direction 
for the mixture ratio a and the mixed region. For example, 
in the event that an object in the actual world 1 which is 
the object of image- taking, is moving as to the sensor 2, 

20 the ratio of signals for the two objects cast into the 
detecting region of the single detecting element of the 
sensor 2 changes in the time direction. The signals for the 
two objects regarding which the ratio changes in the time 
direction, that have been cast into the detecting region of 

25 the single detecting element of the sensor 2, are projected 
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into a single value of the data 3 by the detecting element 
of the sensor 2 . 

The mixture of signals for two objects in the time 
direction due to time integration effects of the sensor 2 
5 will be called time mixture . 

The data continuity detecting unit 101 detects regions 
of pixels in the data 3 where signals of the actual world 1 
for two objects in the actual world 1, for example, have 
been projected. The data continuity detecting unit 101 

10 detects gradient in the data 3 corresponding to the gradient 
of an edge of an image in the actual world 1, for example. 

The actual world estimating unit 102 estimates the 
signals of the actual world 1 by formulating an expression 
using N variables, representing the relation between the 

15 model 161 represented by the N variables and the M pieces of 
data 162, based on the region of the pixels having a 
predetermined mixture ratio a detected by the data 
continuity detecting unit 101 and the gradient of the region, 
for example, and solving the formulated expression. 

20 Description will be made further regarding specific 

estimation of the actual world 1. 

Of the signals of the actual world represented by the 
function F(x, y, z, t)let us consider approximating the 
signals of the actual world represented by the function F(x, 

25 y, t) at the cross-section in the spatial direction Z (the 
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position of the sensor 2) , with an approximation function 
f (x, y, t) determined by a position x in the spatial 
direction X, a position y in the spatial direction Y, and a 
point-in-time t. 
5 Now, the detection region of the sensor 2 has an 

expanse in the spatial direction X and the spatial direction 
Y. In other words, the approximation function f (x, y, t) is 
a function approximating the signals of the actual world 1 
having an expanse in the spatial direction and time 

10 direction, which are acquired with the sensor 2. 

Let us say that projection of the signals of the actual 
world 1 from the sensor 2 yields a value P (x, y, t) of the 
data 3. The value P(x, y, t) of the data 3 is a pixel value 
which the sensor 2 which is an image sensor outputs, for 

15 example. 

Now, in the event that the projection by the sensor 2 
can be formulated, the value obtained by projecting the 
approximation function f (x, y, t) can be represented as a 
projection function S (x, y, t) . 

20 Obtaining the projection function S (x, y, t) has the 

following problems. 

First, generally, the function F (x, y, z, t) 
representing the signals of the actual world 1 can be a 
function with an infinite number of orders. 

25 Second, even if the signals of the actual world could 
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be described as a function, the projection function S (x, y, 
t) via projection of the sensor 2 generally cannot be 
determined. That is to say, the action of projection by the 
sensor 2, in other words, the relation between the input 
5 signals and output signals of the sensor 2, is unknown, so 
the projection function S (x, y, t) cannot be determined. 

With regard to the first problem, let us consider 
expressing the function f (x, y, t) approximating signals of 
the actual world 1 with the sum of products of the function 
10 f±(x, y, t) which is a describable f unction (e . g . , a function 
with a finite number of orders) and variables w it 

Also, with regard to the second problem, formulating 
projection by the sensor 2 allows us to describe the 
function Si(x, y, t) from the description of the function 
15 fi(x, y, t) . 

That is to say, representing the function f (x, y, t) 

approximating signals of the actual world 1 with the sum of 

i 

products of the function f±(x, y, t) and variables wi, 
Expression (5) can be obtained. 

f (x.y.t) = I WjfjCx.y.t) 

i = 1 

20 ... (5) 

For example, as indicated in Expression (6) , the 
relation between the data 3 and the signals of the actual 
world can be formulated as shown in Expression (7) from 
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Expression (5) by formulating the projection of the sensor 2. 

Sj(x, y, t) = fff fj(x, y,t)dxdydt 

... (6) 

N 

PjUj.yj, tj) = I WjSj (x jf y jf tj) 

i =1 

... (7) 

In Expression (7), j represents the index of the data. 

In the event that M data groups (j = 1 through M) 
common with the N variables wi (i = 1 through N) exists in 
Expression (7) , Expression (8) is satisfied, so the model 
161 of the actual world can be obtained from data 3 . 

N ^ M 

... (8) 

N is the number of variables representing the model 161 
approximating the actual world 1. M is the number of pieces 
of data 162 include in the data 3 . 

Representing the function f (x, y, t) approximating the 
actual world 1 signals with Expression (5) allows the 
variable portion wi to be handled independently. At this 
time, i represents the number of variables. Also, the form 
of the function represented by fi can be handed 
independently, and a desired function can be used for fi. 

Accordingly, the number N of the variables w ± can be 
defined without dependence on the function fi, and the 
variables Wi can be obtained from the relation between the 
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number N of the variables w± and the number of pieces of 
data M. 

That is to say, using the following three allows the 
actual world 1 to be estimated from the data 3. 
5 First, the N variables are determined. That is to say, 

Expression (5) is determined. This enables describing the 
actual world 1 using continuity. For example, the signals 
of the actual world 1 can be described with a model 161 
wherein a cross -section is expressed with a polynomial, and 
10 the same cross- sectional shape continues in a constant 
direction . 

Second, for example, projection by the sensor 2 is 
formulated, describing Expression (7) . For example, this is 
formulated such that the results of integration of the 
15 signals of the actual world 2 are data 3. 

Third, M pieces of data 162 are collected to satisfy 
Expression (8) . For example, the data 162 is collected from 
a region having data continuity that has been detected with 
the data continuity detecting unit 101. For example, data 
20 162 of a region wherein a constant cross-section continues, 
which is an example of continuity, is collected. 

In this way, the relation between the data 3 and the 
actual world 1 is described with Expression (5) , and M 
pieces of data 162 are collected, thereby satisfying 
25 Expression (8) , and the actual world 1 can be estimated. 
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More specifically, in the event of N = M, the number of 
variables N and the number of expressions M are equal, so 
the variables w ± can be obtained by formulating a 
simultaneous equation. 

Also, in the event that N < M, various solving methods 
can be applied. For example, the variables w± can be 
obtained by least - square . 

Now, the solving method by least -square will be 
described in detail 

First, an Expression (9) for predicting data 3 from the 

actual world 1 will be shown according to Expression (7) . 

N 

Pj(Xj.yj.tj) = ZwiSiCxj.yj.tj) 

... (9) 

In Expression (9), P ' j (Xj , Yj , tj) is a prediction value. 

The sum of squared differences E for the prediction 
value P f and observed value P is represented by Expression 
(10) . 

E = Z (Pj(x j .y j ,t j )-P , j (x j ,y j .t j )) 2 
j=i 

. . ■ (10) 

The variables wi are obtained such that the sum of 
squared differences E is the smallest. Accordingly, the 
partial differential value of Expression (10) for each 
variable w k is 0 . That is to say, Expression (11) holds. 

|| = -2 I WiSktXj. Yj , tj) (Pj (xj, yj. tj) - I *,S, (xj, yj, tj)) = 0 
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Expression (11) yields Expression (12) . 
M N M 

I (S k (Xj,yj,tj) I WjSj (xj, yj, tj)) = I S k (x jf yj, tj) Pj (xj, yj, tj) 



i=1 



j=1 



. . . (12) 

When Expression (12) holds with K = 1 through N, the 
solution by least -square is obtained. The normal equation 
thereof is shown in Expression (13) . 



M 



\ 



Z s 1 (j)s 1 (j) Z s 1 (j)s 2 (j) Zs 1 (j)s N (j) 

j=i j=i ~ j=i 

I s 2 (j)s 1 {j) Z s 2 (j)s 2 (j) ••• Zs 2 (j}s N (j) 
j=i j=i - j=i 



Z s N (j)s 1 (j) Z s N (j)s 2 (j) Z s N (j)s H (j) 
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/ \ 
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/ H 1 

I S 1 (j)P J (j) 

j=1 

i s 2 (j)Pj(j) 

j=i 



Zs H (j)Pj{j) 

\j=i 



(13) 



Note that in Expression (13), Si (xj , yj , tj) is described 



10 



as S± ( j ) . 
/ 
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. . . (15) 



/ M 

1 Z S 1 (j)P j (j) 



j=1 

M 




j = 1 



I S 2 (j)Pj(j) 



M 



Z s N (j)Pj(j) 



. . . (16) 



From Expression (14) through Expression (16) , 



5 Expression (13) can be expressed as SmatWmat = Pmat- 

In Expression (13) , Si represents the projection of the 
actual world 1. In Expression (13), Pj represents the data 
3. In Expression (13), wi represents variables for 
describing and obtaining the characteristics of the signals 
10 of the actual world 1. 

Accordingly, inputting the data 3 into Expression (13) 
and obtaining Wmat by a matrix solution or the like enables 
the actual world 1 to be estimated. That is to say, the 
actual world 1 can be estimated by computing Expression (17) . 



15 




. . . (17) 



Note that in the event that Smat is not regular, a 



transposed matrix of Smat can be used to obtain What. 
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The actual world estimating unit 102 estimates the 



actual world 1 by, for example, inputting the data 3 into 



Expression (13) and obtaining Wmat by a matrix solution or 



the like. 



5 



Now, an even more detailed example will be described. 



For example, the cross-sectional shape of the signals of the 
actual world 1, i.e., the change in level as to the change 
in position, will be described with a polynomial. Let us 
assume that the cross-sectional shape of the signals of the 

10 actual world 1 is constant, and that the cross- sect ion of 

the signals of the actual world 1 moves at a constant speed. 
Projection of the signals of the actual world 1 from the 
sensor 2 to the data 3 is formulated by three-dimensional 
integration in the time-space direction of the signals of 

15 the actual world 1. 

The assumption that the cross-section of the signals of 
the actual world 1 moves at a constant speed yields 
Expression (18) and Expression (19) . 



dx 
dt 




20 



. . . (18) 



dt Vy 



. . - (19) 



Here, v x and v y are constant. 



- 69 - 



S04P0850 



Using Expression (18) and Expression (19) , the cross- 
sectional shape of the signals of the actual world 1 can be 
represented as in Expression (20) . 



Formulating projection of the signals of the actual 
world 1 from the sensor 2 to the data 3 by three-dimensional 
integration in the time-space direction of the signals of 
the actual world 1 yields Expression (21) . 



In Expression (21), S (x, y, t) represents an integrated 
value the region from position x s to position x e for the 
spatial direction X, from position y s to position y e for the 
spatial direction Y, and from point-in-time t s to point-in- 

15 time t e for the time direction t, i.e., the region 
represented as a space -time cuboid. 

Solving Expression (13) using a desired function f(x', 
y 1 ) whereby Expression (21) can be determined enables the 
signals of the actual world 1 to be estimated. 

20 In the following, we will use the function indicated in 

Expression (22) as an example of the function f(x', y 1 ). 



f (x\ y') = f (x+v x t, y+v y t) 



5 



. . . (20) 




10 



. . . (21) 
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f (x\ y') = w 1 x , +w 2 y , +w 3 

= (x+v x t) +w 2 (y +v y t) +w 3 

...(22) 

That is to say, the signals of the actual world 1 are 
estimated to include the continuity represented in 
Expression (18) , Expression (19) , and Expression (22) . This 
indicates that the cross-section with a constant shape is 
moving in the space- time direction as shown in Fig. 18. 

Substituting Expression (22) into Expression (21) 
yields Expression (23) . 

S(x.y.t) = / x Xe / ye / t te f(x+v x t,y+v y t)dxdydt 

s * s s 

= Vo I ume ( -y- ( x e +x s +v x ( t e +t s ) ) 

+ X (ye+ys+v y (te+ts))+W2) 

= w 0 S 0 (x, y, t) +wj S 1 (x, y, t) +w 2 S 2 (x, y, t) 

. . . (23) 

wherein 

Volume = (x e - x s ) (y e ~ y s ) (t e -t s ) 

5 0 (x, y, t) = Volume/2 x (x e + x s + v x (t e + t s ) ) 

51 (x, y, t) = Volume/2 x (y e + y s + v y (t e + t s ) ) 

5 2 (x, y, t) = 1 
holds . 

Fig. 19 is a diagram illustrating an example of the M 
pieces of data 162 extracted from the data 3. For example, 
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let us say that 27 pixel values are extracted as the data 
162, and that the extracted pixel values are Pj (x, y, t) . 
In this case, j is 0 through 26. 

In the example shown in Fig. 19, in the event that the 
5 pixel value of the pixel corresponding to the position of 
interest at the point-in- time t which is n is P 13 (x, y, t) , 
and the direction of array of the pixel values of the pixels 
having the continuity of data (e.g., the direction in which 
the same- shaped pawl shapes detected by the data continuity 

10 detecting unit 101 are arrayed) is a direction connecting 

P 4 (x, y, t) , Pi 3 (x, y, t) , and P 22 (x, y, t), the pixel values 
P9(x, y, t) through P 17 (x, y, t) at the point-in-time t which 
is n, the pixel values P 0 (x, y, t) through P 8 (x, y, t) at the 
point-in-time t which is n - 1 which is earlier in time than 

15 n, and the pixel values Pi 8 (x, y, t) through P 2 e(x, y, t) at 
the point-in-time t which is n + 1 which is later in time 
than n, are extracted. 

Now, the region regarding which the pixel values, which 
are the data 3 output from the image sensor which is the 

20 sensor 2, have been obtained, have a time-direction and two- 
dimensional spatial direction expansion. Now, for example, 
the center of gravity of the cuboid corresponding to the 
pixel values (the region regarding which the pixel values 
have been obtained) can be used as the position of the pixel 

25 in the space-time direction. 
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Generating Expression (13) from the 27 pixel values P 0 (x, 
y, t) through P 26 (x, y, t) and from Expression (23), and 
obtaining W, enables the actual world 1 to be estimated. 
In this way, the actual world estimating unit 102 
5 generates Expression (13) from the 27 pixel values P 0 (x, y, 
t) through P 26 (x, y, t) and from Expression (23), and obtains 
W, thereby estimating the signals of the actual world 1. 

Note that a Gaussian function, a sigmoid function, or 
the like, can be used for the function f±(x, y, t) . 
10 An example of processing for generating high-resolution 

data with even higher resolution, corresponding to the data 
3, from the estimated actual world 1 signals, will be 
described with reference to Fig. 20 through Fig. 23. 

As shown in Fig. 20, the data 3 has a value wherein 
15 signals of the actual world 1 are integrated in the time 
direction and two-dimensional spatial directions. For 
example, a pixel value which is data 3 that has been output 
from the image sensor which is the sensor 2 has a value 
wherein the signals of the actual world 1, which is light 
2 0 cast into the detecting device, are integrated by the 
shutter time which is the detection time in the time 
direction, and integrated by the photorecept ion region of 
the detecting element in the spatial direction. 

Conversely, as shown in Fig. 21, the high-resolution 
25 data with even higher resolution in the spatial direction is 
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generated by integrating the estimated actual world 1 
signals in the time direction by the same time as the 
detection time of the sensor 2 which has output the data 3, 
and also integrating in the spatial direction by a region 
5 narrower in comparison with the photoreception region of the 
detecting element of the sensor 2 which has output the data 
3 . 

Note that at the time of generating the high-resolution 
data with even higher resolution in the spatial direction, 

10 the region where the estimated signals of the actual world 1 
are integrated can be set completely disengaged from 
photoreception region of the detecting element of the sensor 
2 which has output the data 3. For example, the high- 
resolution data can be provided with resolution which is 

15 that of the data 3 magnified in the spatial direction by an 
integer, of course, and further, can be provided with 
resolution which is that of the data 3 magnified in the 
spatial direction by a rational number such as 5/3 times, 
for example. 

20 Also, as shown in Fig. 22, the high- resolution data 

with even higher resolution in the time direction is 
generated by integrating the estimated actual world 1 
signals in the spatial direction by the same region as the 
photoreception region of the detecting element of the sensor 

25 2 which has output the data 3, and also integrating in the 
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time direction by a time shorter than the detection time of 
the sensor 2 which has output the data 3 . 

Note that at the time of generating the high-resolution 
data with even higher resolution in the time direction, the 
5 time by which the estimated signals of the actual world 1 

are integrated can be set completely disengaged from shutter 
time of the detecting element of the sensor 2 which has 
output the data 3. For example, the high-resolution data 
can be provided with resolution which is that of the data 3 

10 magnified in the time direction by an integer, of course, 
and further, can be provided with resolution which is that 
of the data 3 magnified in the time direction by a rational 
number such as 7/4 times, for example. 

High-resolution data with movement blurring removed is 

15 generated by integrating the estimated actual world 1 

signals only in the spatial direction and not in the time 
direction . 

Further, as shown in Fig. 23, high-resolution data with 
higher resolution in the time direction and space direction 

20 is generated by integrating the estimated actual world 1 
signals in the spatial direction by a region narrower in 
comparison with the photorecept ion region of the detecting 
element of the sensor 2 which has output the data 3, and 
also integrating in the time direction by a time shorter in 

25 comparison with the detection time of the sensor 2 which has 
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output the data 3 . 

In this case, the region and time for integrating the 
estimated actual world 1 signals can be set completely- 
unrelated to the photoreception region and shutter time of 
5 the detecting element of the sensor 2 which has output the 
data 3 . 

Thus, the image generating unit 103 generates data with 
higher resolution in the time direction or the spatial 
direction, by integrating the estimated actual world 1 

10 signals by a desired space-time region, for example. 

Accordingly, data which is more accurate with regard to 
the signals of the actual world 1, and which has higher 
resolution in the time direction or the space direction, can 
be generated by estimating the signals of the actual world 1. 

15 Fig. 24 through Fig. 2 8 illustrate an example of an 

input image and an example of the results of processing with 
the signal processing device 4 used for signal processing. 

Fig. 24 is a diagram illustrating an original image of 
an input image (equivalent to light signals of the actual 

20 world 1) . Fig. 25 is a diagram illustrating an example of 
an input image. The input image shown in Fig. 25 is an 
image generated by taking the average value of pixel values 
of pixels belonging to blocks made up of 2 by 2 pixels of 
the image shown in Fig. 24, as the pixel value of a single 

25 pixel. That is to say, the input image is an image obtained 
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by applying spatial direction integration to the image shown 
in Fig. 24, imitating the integrating properties of the 
sensor . 

The original image shown in Fig. 24 contains an image 
5 of a fine line inclined at approximately 5 degrees in the 
clockwise direction from the vertical direction. In the 
same way, the input image shown in Fig. 2 5 contains an image 
of a fine line inclined at approximately 5 degrees in the 
clockwise direction from the vertical direction. 

10 Fig. 26 is a diagram illustrating an image obtained by 

applying conventional class classification adaptation 
processing to the input image shown in Fig. 25. Now, class 
classification processing is made up of class classification 
processing and adaptation processing, wherein the data is 

15 classified based on the nature thereof by the class 

classification adaptation processing, and subjected to 
adaptation processing for each class. In the adaptation 
processing, a low-image quality or standard image quality 
image, for example, is converted into a high image quality 

2 0 image by being subjected to mapping (mapping) using a 
predetermined tap coefficient. 

That is to say, with adaptation processing, first data 
is converted into second data by being mapped (mapping) 
using a predetermined tap coefficient. 

25 Now, adaptation processing will be described with 
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regard to a mapping method using the tap coefficient, 

wherein for example, a linear combination model is used, and 

also a low-resolution image or standard resolution SD 

(Standard Definition) image obtained by filtering a high- 

5 resolution HD (High Definition) image with a low-pass filter 

is used as the first data, and the HD image used for 

obtaining the SD image is used as the second data. 

Now, under the above -described conditions, an HD pixel 

y making up the HD image can be obtained with the following 

10 linear Expression (linear combination) using multiple SD 

pixels extracted from SD pixels making up the SD image, as a 

prediction tap for predicting the HD image, and the tap 

coefficient . 
N 

y=Iw n X n 
n=1 

. - . (24) 

15 Wherein, in Expression (24) , x n represents the pixel 

value of the n ■ th pixel in the SD image, making up a 
prediction tap regarding the HD pixel y, and w n represents 
the n'th tap coefficient to be multiplied by (the pixel 
value of) the n'th SD pixel. Note that in Expression (24), 

2 0 a prediction tap is made up of an N number of SD pixels x x , 
x 2 , ■ • • , x N . 

Now, the pixel value y of the HD pixel can be obtained 
by a quadratic expression or higher, instead of the linear 
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expression shown in Expression (24) . 

Now, in the HD image, saying that y k represents the true 
value of (the pixel value of) the k ■ th HD pixel, and y k ' 
represents a prediction value of the true value y k obtained 
by Expression (24) , the prediction error e k thereof is as 
expressed in the following Expression, for example. 



The prediction value y k ' in Expression (25) is obtained 
according to Expression (24) , so substituting the y k ' in 
Expression (25) according Expression (24) yields the 
following Expression . 



Wherein, in Expression (26), X n , k represents the n'th SD 
pixel making up the prediction tap for the k'th HD pixel. 

While a tap coefficient w n wherein the prediction error 
e k is 0 in Expression (26) is optimal for predicting the HD 
pixel, obtaining such a tap coefficient w n for all HD pixels 
is generally difficult. 

Accordingly, as a rule representing the tap coefficient 
w n as being optimal, employing the least -square method for 
example, means that the optimal tap coefficient w n can be 
obtained by minimizing the summation E of squared errors 



ek^Vk-yk' 



. . . (25) 




. . . (26) 
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represented in the following Expression for example, as a 
statistical error. 

E=Ie k 2 

k=1 

. . . (27) 

Wherein, in Expression (27) , K represents the number of 
samples of sets made up of an HD pixel y k and SD pixels Xi, k , 
X 2 ,k, " • ' / X N;k making up a prediction tap regarding that HD 
pixel y k . 

The tap coefficient W n which makes the summation E of 
squared errors in Expression (27) smallest (the minimum) is 
such that the partial differentiation of the summation E by 
the tap coefficient w n yields 0. Accordingly, the following 
Expression must be satisfied. 

-^=p 1 -^-+eo-^- + ---+eu-^-=0 (n=1 2 ••• N) 

3W n e1 3W n +e2 3W n + +<3k 9w n U ^ ' ' Z ' ' W 

. . . (28) 

Now, partial differentiation of the above Expression 
(26) by the tap coefficient w n yields the following 
Expression. 

. . . (29) 

Expressions (28) and (29) yield the following 
Expression. 

k k k 

Ze k xi, k =0, Ze k X2,k=0, ■■•IekX N ,k=0 

k=1 k=1 k=1 
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. . . (30) 

Substituting Expression (26) for e k in Expression (30) 
allows Expression (30) to be expressed in the form of the 



normal equation in Expression (31) . 

k k k 

(IXl.kXl.k) (£Xl,kX2,k) ••- (ZXl, k XN,k) 

k=1 k=1 k=1 

k k k 

(IX 2 ,kXl,k) (ZX2,kX2,k) •• (ZX2,kXN,k) 

k=1 k=1 k=1 



k k k 

(ZXN,kXl,k) (ZXN.kX2,k) • • (ZXN,kXN.k) 
k=1 k=1 k=1 



W1 



W2 



WN 



(ZXl.kYk) 

k=1 



( Z X2, kYk) 

k=1 



(Zxn, kYk) 

k=1 



(31) 



Preparing a certain number of sets of the HD pixel y k 
and SD pixels x njk allows as many of the normal equations in 
Expression (31) to be formulated as the number of tap 

10 coefficients w n to be obtained, and solving Expression (31) 
allows optimal tap coefficients w n to be obtained. Note 
that sweeping (Gauss- Jordan elimination) or the like, for 
example, can be used for solving Expression (31) . 

As described above, adaptation processing involves 

15 taking a great number of HD pixels y 1# y 2 , ■ • •, Yk as tutor 
data to serve as a tutor for learning for tap coefficients, 
and also taking SD pixels Xi, k , X 2 , k / ' ' ' > X N>k making up a 
prediction tap regarding each HD pixel y k as student data to 
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serve as a student for learning for tap coefficients, and 
solving Expression (31) , thereby performing learning for 
obtaining an optimal tap coefficient w n/ and further using 
the optimal tap coefficient w n to perform mapping 
5 (conversion) of an SD image into an HD image according to 
Expression (24) . 

Now, for the SD pixels X lfk/ X 2 , k , ■ • • # X N/k making up the 
prediction tap regarding the HD pixels y k/ an SD pixel close 
to a position on the SD image corresponding to the HD pixel 

10 y k/ either spatially or temporally, can be used. 

Also, with class classification adaptation processing, 
learning of the tap coefficient w n and mapping using the tap 
coefficient w n are performed by class. With class 
classification adaptation processing, class classification 

15 processing is performed with regard to the HD pixel y k of 

interest, and learning of the tap coefficient w n and mapping 
using the tap coefficient w n are performed for each class 
obtained by the class classification processing. 

An example of class classification processing with 

20 regard to the HD pixel y k is to extract multiple SD pixels 
from the SD image to serve as a class tap used for class 
classification of the HD pixel y k , and to perform M-bit ADRC 
(Adaptive Dynamic Range Coding) using the class tap made up 
of the multiple SD pixels. 

25 In M-bit ADRC processing, the maximum value MAX and 
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minimum value MIN of the SD pixels making up the class tap 
are detected, DR = MAX - MIN is set as a local dynamic range, 
and the SD pixels making up the class tap are re-quantized 
into K bits, based on this dynamic range DR. That is to say, 
5 the minimum value MIN is subtracted from the SD pixels 
making up the class tap, and the subtraction value is 
divided by DR/2 K (quantized) . Accordingly, in the event 
that a class tap is subjected to 1-bit ADRC processing for 
example, each SD pixel making up the class tap is one bit. 

10 In this case, a bit string obtained by arraying in a 

predetermined order the 1-bit pixel values regarding each SD 
pixel making up the class tap that have been obtained as 
described above is output as ADRC code, and the ADRC code is 
taken as a class code representing the class. 

15 Note that class classification adaptation processing 

differs from simple interpolation processing or the like, 
for example, in that components not included in the SD image 
but are included in the HD image are reproduced. That is to 
say, with class classification adaptation processing, it 

20 would seem by looking at Expression (24) alone that this is 
the same as interpolation processing using a so-called 
interpolation filter, but the tap coefficient w n which is 
equivalent to the tap coefficient of the interpolation 
filter has been obtained by learning using an HD image 

2 5 serving as tutor data and an SD image serving as student 
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data, so the component contained in the HD image can be 
reproduced . 

Now, tap coefficients w n which perform various types of 
conversion can be obtained in the tap coefficient w n 
5 learning, depending on what sort of combination of tutor 
data y and student data x is employed. 

That is to say, in a case of taking a high-resolution 
HD image as the tutor data y and taking an SD image wherein 
the resolution of the HD image has been deteriorated as the 
10 student data x, for example, a tap coefficient w n for 

mapping an image into an image with the resolution improved, 
can be obtained. Further, in a case of taking an HD image 
as the tutor data y and taking an SD image wherein the 
number of pixels of the HD image has been reduced as the 
15 student data x, for example, a tap coefficient w n for 

mapping an image into an image with an increased number of 
pixels making up the image, can be obtained. 

Fig. 2 6 is an image obtained by subjecting the input 
image shown in Fig. 2 5 to mapping by the class 
20 classification adaptation processing such as described above. 
It can be understood in the image shown in Fig. 2 6 that the 
image of the fine line is different to that of the original 
image in Fig. 24. 

Fig. 27 is a diagram illustrating the results of 
25 detecting the fine line regions from the input image shown 
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in the example in Fig. 25, by the data continuity detecting 
unit 101. In Fig. 27, the white region indicates the fine 
line region, i.e., the region wherein the arc shapes shown 
in Fig. 10 are arrayed. 
5 Fig. 28 is a diagram illustrating an example of the 

output image obtained by performing signal processing at the 
signal processing device 4, with the image shown in Fig. 2 5 
as the input image. As shown in Fig. 28, the signals 
processing device 4 yields an image closer to the fine line 

10 image of the original image shown in Fig. 24. 

Fig. 29 is a flowchart for describing the processing of 
signals with the signal processing device 4. 

In step S101, the data continuity detecting unit 101 
executes the processing for detecting continuity. The data 

15 continuity detecting unit 101 detects data continuity 
contained in the input image which is the data 3, and 
supplies the data continuity information indicating the 
detected data continuity to the actual world estimating unit 
102 and the image generating unit 103. 

20 The data continuity detecting unit 101 detects the 

continuity of data corresponding to the continuity of the 
signals of the actual world. In the processing in step S101, 
the continuity of data detected by the data continuity 
detecting unit 101 is either part of the continuity of the 

25 image of the actual world 1 contained in the data 3, or 
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continuity which has changed from the continuity of the 
signals of the actual world 1. 

For example, the data continuity detecting unit 101 
detects the data continuity by detecting a region having a 
5 continuity in a predetermined dimensional direction. Also, 
for example, the data continuity detecting unit 101 detects 
data continuity by detecting angle (gradient) in the spatial 
direction indicating the array of the same shape. 

Details of the continuity detecting processing in step 

10 S101 will be described later. 

Note that the data continuity information can be used 
as features, indicating the characteristics of the data 3. 

In step S102, the actual world estimating unit 102 
executes processing for estimating the actual world. That 

15 is to say, the actual world estimating unit 102 estimates 

the signals of the actual world based on the input image and 
the data continuity information supplied from the data 
continuity detecting unit 101. In the processing in step 
S102 for example, the actual world estimating unit 102 

20 estimates the signals of the actual world 1 by predicting a 
model 161 approximating (describing) the actual world 1 . 
The actual world estimating unit 102 supplies the actual 
world estimation information indicating the estimated 
signals of the actual world 1 to the image generating unit 

25 103. 
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For example, the actual world estimating unit 102 
estimates the actual world 1 signals by predicting the width 
of the linear object. Also, for example, the actual world 
estimating unit 102 estimates the actual world 1 signals by 
5 predicting a level indicating the color of the linear object. 

Details of processing for estimating the actual world 
in step S102 will be described later. 

Note that the actual world estimation information can 
be used as features, indicating the characteristics of the 
10 data 3. 

In step S103, the image generating unit 103 performs 
image generating processing, and the processing ends. That 
is to say, the image generating unit 103 generates an image 
based on the actual world estimation information, and 

15 outputs the generated image. Or, the image generating unit 
103 generates an image based on the data continuity 
information and actual world estimation information, and 
outputs the generated image. 

For example, in the processing in step S103, the image 

20 generating unit 103 integrates the estimated real world 

light in the spatial direction, based on the actual world 
estimated information, thereby generating an image with 
higher resolution in the spatial direction in comparison 
with the input image, and outputs the generated image. For 

25 example, the image generating unit 103 integrates estimated 
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real world light in the time-space direction, based on the 
actual world estimated information, hereby generating an 
image with higher resolution in the time direction and the 
spatial direction in comparison with the input image, and 
5 outputs the generated image. The details of the image 

generating processing in step S103 will be described later. 

Thus, the signal processing device 4 detects data 
continuity from the data 3, and estimates the actual world 1 
from the detected data continuity. The signal processing 

10 device 4 then generates signals closer approximating the 
actual world 1 based on the estimated actual world 1. 

As described above, in the event of performing the 
processing for estimating signals of the real world, 
accurate and highly-precise processing results can be 

15 obtained. 

Also, in the event that first signals which are real 
world signals having first dimensions are projected, the 
continuity of data corresponding to the lost continuity of 
the real world signals is detected for second signals of 

2 0 second dimensions, having a number of dimensions fewer than 
the first dimensions, from which a part of the continuity of 
the signals of the real world has been lost, and the first 
signals are estimated by estimating the lost real world 
signals continuity based on the detected data continuity, 

25 accurate and highly-precise processing results can be 
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obtained as to the events in the real world. 

Next, the details of the configuration of the data 
continuity detecting unit 101 will be described. 

Fig. 30 is a block diagram illustrating the 
configuration of the data continuity detecting unit 101. 

Upon taking an image of an object which is a fine line, 
the data continuity detecting unit 101, of which the 
configuration is shown in Fig. 30, detects the continuity of 
data contained in the data 3, which is generated from the 
continuity in that the cross- sectional shape which the 
object has is the same. That is to say, the data continuity 
detecting unit 101 of the configuration shown in Fig. 30 
detects the continuity of data contained in the data 3, 
which is generated from the continuity in that the change in 
level of light as to the change in position in the direction 
orthogonal to the length-wise direction is the same at an 
arbitrary position in the length-wise direction, which the 
image of the actual world 1 which is a fine line, has. 

More specifically, the data continuity detecting unit 
101 of which configuration is shown in Fig. 30 detects the 
region where multiple arc shapes (half-disks) having a 
predetermined length are arrayed in a diagonally-offset 
adjacent manner, within the data 3 obtained by taking an 
image of a fine line with the sensor 2 having spatial 
integration effects . 
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The data continuity detecting unit 101 extracts the 
portions of the image data (hereafter referred to as con- 
continuity component) other than the portion of the image 
data where the image of the fine line having data continuity 
5 has been projected (hereafter, the portion of the image data 
where the image of the fine line having data continuity has 
been projected will also be called continuity component, and 
the other portions will be called non- continuity component) , 
from an input image which is the data 3, detects the pixels 

10 where the image of the fine line of the actual world 1 has 
been projected, from the extracted non- continuity component 
and the input image, and detects the region of the input 
image made up of pixels where the image of the fine line of 
the actual world 1 has been projected. 

15 a non -continuity component extracting unit 201 extracts 

the non-continuity component from the input image, and 
supplies the non- continuity component information indicating 
the extracted non-continuity component to a peak detecting 
unit 2 02 and a monotonous increase/decrease detecting unit 

20 203 along with the input image. 

For example, as shown in Fig. 31, in the event that an 
image of the actual world 1 wherein a fine line exists in 
front of a background with an approximately constant light 
level is projected on the data 3, the non- continuity 

25 component extracting unit 201 extracts the non-continuity 
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component which is the background, by approximating the 
background in the input image which is the data 3, on a 
plane, as shown in Fig. 32. In Fig. 32, the solid line 
indicates the pixel values of the data 3, and the dotted 
5 line illustrates the approximation values indicated by the 
plane approximating the background. In Fig. 32, A denotes 
the pixel value of the pixel where the image of the fine 
line has been projected, and the PL denotes the plane 
approximating the background. 
10 In this way, the pixel values of the multiple pixels at 

the portion of the image data having data continuity are 
discontinuous as to the non- continuity component. 

The non-continuity component extracting unit 201 
detects the discontinuous portion of the pixel values of the 
15 multiple pixels of the image data which is the data 3, where 
an image which is light signals of the actual world 1 has 
been projected and a part of the continuity of the image of 
the actual world 1 has been lost . 

Details of the processing for extracting the non- 
20 continuity component with the non-continuity component 
extracting unit 2 01 will be described later. 

The peak detecting unit 2 02 and the monotonous 
increase/decrease detecting unit 2 03 remove the non- 
continuity component from the input image, based on the non- 
25 continuity component information supplied from the non- 
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continuity component extracting unit 201. For example, the 
peak detecting unit 202 and the monotonous increase/decrease 
detecting unit 203 remove the non-continuity component from 
the input image by setting the pixel values of the pixels of 
5 the input image where only the background image has been 

projected, to 0. Also, for example, the peak detecting unit 
2 02 and the monotonous increase/decrease detecting unit 2 03 
remove the non-continuity component from the input image by 
subtracting values approximated by the plane PL from the 

10 pixel values of each pixel of the input image. 

Since the background can be removed from the input 
image, the peak detecting unit 2 02 through continuousness 
detecting unit 2 04 can process only the portion of the image 
data where the fine line has be projected, thereby further 

15 simplifying the processing by the peak detecting unit 202 
through the continuousness detecting unit 204. 

Note that the non-continuity component extracting unit 
201 may supply image data wherein the non-continuity 
component has been removed from the input image, to the peak 

2 0 detecting unit 2 02 and the monotonous increase/decrease 
detecting unit 203. 

In the example of processing described below, the image 
data wherein the non-continuity component has been removed 
from the input image, i.e., image data made up from only 

25 pixel containing the continuity component, is the object. 
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Now, description will be made regarding the image data 
upon which the fine line image has been projected, which the 
peak detecting unit 2 02 through continuousness detecting 
unit 204 are to detect. 
5 In the event that there is no optical LPF, the cross- 

dimensional shape in the spatial direction Y (change in the 
pixel values as to change in the position in the spatial 
direction) of the image data upon which the fine line image 
has been projected as shown in Fig. 31 can be thought to be 

10 the trapezoid shown in Fig. 33, or the triangle shown in Fig. 
34, from the spatial; integration effects of the image 
sensor which is the sensor 2. However, ordinary image 
sensors have an optical LPF with the image sensor obtaining 
the image which has passed through the optical LPF and 

15 projects the obtained image on the data 3, so in reality, 
the cross -dimensional shape of the image data with fine 
lines in the spatial direction Y has a shape resembling 
Gaussian distribution, as shown in Fig. 35. 

The peak detecting unit 2 02 through continuousness 

2 0 detecting unit 2 04 detect a region made up of pixels upon 
which the fine line image has been projected wherein the 
same cross-sectional shape (change in the pixel values as to 
change in the position in the spatial direction) is arrayed 
in the vertical direction in the screen at constant 

25 intervals, and further, detect a region made up of pixels 
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upon which the fine line image has been projected which is a 
region having data continuity, by detecting regional 
connection corresponding to the length-wise direction of the 
fine line of the actual world 1. That is to say, the peak 
5 detecting unit 202 through continuousness detecting unit 2 04 
detect regions wherein arc shapes (half-disc shapes) are 
formed on a single vertical row of pixels in the input image, 
and determine whether or not the detected regions are 
adjacent in the horizontal direction, thereby detecting 

10 connection of regions where arc shapes are formed, 

corresponding to the length-wise direction of the fine line 
image which is signals of the actual world 1. 

Also, the peak detecting unit 2 02 through 
continuousness detecting unit 2 04 detect a region made up of 

15 pixels upon which the fine line image has been projected 
wherein the same cross -sectional shape is arrayed in the 
horizontal direction in the screen at constant intervals, 
and further, detect a region made up of pixels upon which 
the fine line image has been projected which is a region 

20 having data continuity, by detecting connection of detected 
regions corresponding to the length-wise direction of the 
fine line of the actual world 1. That is to say, the peak 
detecting unit 2 02 through continuousness detecting unit 2 04 
detect regions wherein arc shapes are formed on a single 

2 5 horizontal row of pixels in the input image, and determine 
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whether or not the detected regions are adjacent in the 
vertical direction, thereby detecting connection of regions 
where arc shapes are formed, corresponding to the length- 
wise direction of the fine line image, which is signals of 
5 the actual world 1 . 

First, description will be made regarding processing 
for detecting a region of pixels upon which the fine line 
image has been projected wherein the same arc shape is 
arrayed in the vertical direction in the screen at constant 

10 intervals. 

The peak detecting unit 2 02 detects a pixel having a 
pixel value greater than the surrounding pixels, i.e., a 
peak, and supplies peak information indicating the position 
of the peak to the monotonous increase/decrease detecting 

15 unit 203. In the event that pixels arrayed in a single 
vertical direction row in the screen are the object, the 
peak detecting unit 202 compares the pixel value of the 
pixel position upwards in the screen and the pixel value of 
the pixel position downwards in the screen, and detects the 

2 0 pixel with the greater pixel value as the peak. The peak 
detecting unit 202 detects one or multiple peaks from a 
single image, e.g., from the image of a single frame. 

A single screen contains frames or fields. This holds 
true in the following description as well. 

25 For example, the peak detecting unit 202 selects a 
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pixel of interest from pixels of an image of one frame which 
have not yet been taken as pixels of interest, compares the 
pixel value of the pixel of interest with the pixel value of 
the pixel above the pixel of interest, compares the pixel 
5 value of the pixel of interest with the pixel value of the 
pixel below the pixel of interest, detects a pixel of 
interest which has a greater pixel value than the pixel 
value of the pixel above and a greater pixel value than the 
pixel value of the pixel below, and takes the detected pixel 

10 of interest as a peak. The peak detecting unit 202 supplies 
peak information indicating the detected peak to the 
monotonous increase/decrease detecting unit 203. 

There are cases wherein the peak- detecting unit 202 
does not detect a peak. For example, in the event that the 

15 pixel values of all of the pixels of an image are the same 

value, or in the event that the pixel values decrease in one 
or two directions, no peak is detected. In this case, no 
fine line image has been projected on the image data. 

The monotonous increase/decrease detecting unit 2 03 

2 0 detects a candidate for a region made up of pixels upon 
which the fine line image has been projected wherein the 
pixels are vertically arrayed in a single row as to the peak 
detected by the peak detecting unit 2 02, based upon the peak 
information indicating the position of the peak supplied 

25 from the peak detecting unit 202, and supplies the region 
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information indicating the detected region to the 
continuousness detecting unit 2 04 along with the peak 
information. 

More specifically, the monotonous increase/decrease 
5 detecting unit 2 03 detects a region made up of pixels having 
pixel values monotonously decreasing with reference to the 
peak pixel value, as a candidate of a region made up of 
pixels upon which the image of the fine line has been 
projected. Monotonous decrease means that the pixel values 
10 of pixels which are farther distance-wise from the peak are 
smaller than the pixel values of pixels which are closer to 
the peak. 

Also, the monotonous increase/decrease detecting unit 
2 03 detects a region made up of pixels having pixel values 

15 monotonously increasing with reference to the peak pixel 
value, as a candidate of a region made up of pixels upon 
which the image of the fine line has been projected. 
Monotonous increase means that the pixel values of pixels 
which are farther distance-wise from the peak are greater 

2 0 than the pixel values of pixels which are closer to the peak. 
In the following, the processing regarding regions of 
pixels having pixel values monotonously increasing is the 
same as the processing regarding regions of pixels having 
pixel values monotonously decreasing, so description thereof 

25 will be omitted. Also, with the description regarding 
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processing for detecting a region of pixels upon which the 
fine line image has been projected wherein the same arc 
shape is arrayed horizontally in the screen at constant 
intervals, the processing regarding regions of pixels having 
5 pixel values monotonously increasing is the same as the 

processing regarding regions of pixels having pixel values 
monotonously decreasing, so description thereof will be 
omitted. 

For example, the monotonous increase/decrease detecting 

10 unit 203 obtains pixel values of each of the pixels in a 
vertical row as to a peak, the difference as to the pixel 
value of the pixel above, and the difference as to the pixel 
value of the pixel below. The monotonous increase/decrease 
detecting unit 203 then detects a region wherein the pixel 

15 value monotonously decreases by detecting pixels wherein the 
sign of the difference changes. 

Further, the monotonous increase/decrease detecting 
unit 203 detects, from the region wherein pixel values 
monotonously decrease, a region made up of pixels having 

2 0 pixel values with the same sign as that of the pixel value 

of the peak, with the sign of the pixel value of the peak as 
a reference, as a candidate of a region made up of pixels 
upon which the image of the fine line has been projected. 

For example, the monotonous increase/decrease detecting 

25 unit 203 compares the sign of the pixel value of each pixel 
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with the sign of the pixel value of the pixel above and sign 
of the pixel value of the pixel below, and detects the pixel 
where the sign of the pixel value changes, thereby detecting 
a region of pixels having pixel values of the same sign as 
5 the peak within the region where pixel values monotonously 
decrease . 

Thus, the monotonous increase/decrease detecting unit 
203 detects a region formed of pixels arrayed in a vertical 
direction wherein the pixel values monotonously decrease as 
10 to the peak and have pixels values of the same sign as the 
peak. 

Fig. 3 6 is a diagram describing processing for peak 
detection and monotonous increase/decrease region detection, 
for detecting the region of pixels wherein the image of the 
15 fine line has been projected, from the pixel values as to a 
position in the spatial direction Y. 

In Fig. 36 through Fig. 38, P represents a peak. In 
the description of the data continuity detecting unit 101 of 
which the configuration is shown in Fig. 30, P represents a 
2 0 peak. 

The peak detecting unit 2 02 compares the pixel values 
of the pixels with the pixel values of the pixels adjacent 
thereto in the spatial direction Y, and detects the peak P 
by detecting a pixel having a pixel value greater than the 
25 pixel values of the two pixels adjacent in the spatial 
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direction Y. 

The region made up of the peak P and the pixels on both 
sides of the peak P in the spatial direction Y is a 
monotonous decrease region wherein the pixel values of the 
5 pixels on both sides in the spatial direction Y monotonously 
decrease as to the pixel value of the peak P. In Fig. 36, 
the arrow denoted A and the arrow denoted by B represent the 
monotonous decrease regions existing on either side of the 
peak P. 

10 The monotonous increase/decrease detecting unit 2 03 

obtains the difference between the pixel values of each 
pixel and the pixel values of the pixels adjacent in the 
spatial direction Y, and detects pixels where the sign of 
the difference changes. The monotonous increase/decrease 

15 detecting unit 2 03 takes the boundary between the detected 

pixel where the sign of the difference changes and the pixel 
immediately prior thereto (on the peak P side) as the 
boundary of the fine line region made up of pixels where the 
image of the fine line has been projected. 

2 0 In Fig. 36, the boundary of the fine line region which 

is the boundary between the pixel where the sign of the 
difference changes and the pixel immediately prior thereto 
(on the peak P side) is denoted by C. 

Further, the monotonous increase/decrease detecting 

25 unit 2 03 compares the sign of the pixel values of each pixel 
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with the pixel values of the pixels adjacent thereto in the 
spatial direction Y, and detects pixels where the sign of 
the pixel value changes in the monotonous decrease region. 
The monotonous increase/decrease detecting unit 2 03 takes 
5 the boundary between the detected pixel where the sign of 
the pixel value changes and the pixel immediately prior 
thereto (on the peak P side) as the boundary of the fine 
line region. 

In Fig. 36, the boundary of the fine line region which 
10 is the boundary between the pixel where the sign of the 

pixel value changes and the pixel immediately prior thereto 
(on the peak P side) is denoted by D. 

As shown in Fig. 36, the fine line region F made up of 
pixels where the image of the fine line has been projected 
15 is the region between the fine line region boundary C and 
the fine line region boundary D. 

The monotonous increase/decrease detecting unit 203 
obtains a fine line region F which is longer than a 
predetermined threshold value, from fine line regions F made 
20 up of such monotonous increase/decrease regions, i.e., a 
fine line region F having a greater number of pixels than 
the threshold value. For example, in the event that the 
threshold value is 3, the monotonous increase/decrease 
detecting unit 2 03 detects a fine line region F including 4 
25 or more pixels. 
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Further, the monotonous increase/decrease detecting 
unit 2 03 compares the pixel value of the peak P, the pixel 
value of the pixel to the right side of the peak P, and the 
pixel value of the pixel to the left side of the peak P, 
5 from the fine line region F thus detected, each with the 
threshold value, detects a fine line pixel region F having 
the peak P wherein the pixel value of the peak P exceeds the 
threshold value, and wherein the pixel value of the pixel to 
the right side of the peak P is the threshold value or lower, 
10 and wherein the pixel value of the pixel to the left side of 
the peak P is the threshold value or lower, and takes the 
detected fine line region F as a candidate for the region 
made up of pixels containing the component of the fine line 
image . 

15 in other words, determination is made that a fine line 

region F having the peak P, wherein the pixel value of the 
peak P is the threshold value or lower, or wherein the pixel 
value of the pixel to the right side of the peak P exceeds 
the threshold value, or wherein the pixel value of the pixel 

2 0 to the left side of the peak P exceeds the threshold value, 
does not contain the component of the fine line image, and 
is eliminated from candidates for the region made up of 
pixels including the component of the fine line image. 
That is, as shown in Fig. 37, the monotonous 

25 increase/decrease detecting unit 203 compares the pixel 
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value of the peak P with the threshold value, and also 
compares the pixel value of the pixel adjacent to the peak P 
in the spatial direction X (the direction indicated by the 
dotted line AA') with the threshold value, thereby detecting 
5 the fine line region F to which the peak P belongs, wherein 
the pixel value of the peak P exceeds the threshold value 
and wherein the pixel values of the pixel adjacent thereto 
in the spatial direction X are equal to or below the 
threshold value. 

10 Fig. 38 is a diagram illustrating the pixel values of 

pixels arrayed in the spatial direction X indicated by the 
dotted line AA' in Fig. 37. The fine line region F to which 
the peak P belongs, wherein the pixel value of the peak P 
exceeds the threshold value Th s and wherein the pixel values 

15 of the pixel adjacent thereto in the spatial direction X are 
equal to or below the threshold value Th s/ contains the fine 
line component. 

Note that an arrangement may be made wherein the 
monotonous increase/decrease detecting unit 2 03 compares the 

20 difference between the pixel value of the peak P and the 
pixel value of the background with the threshold value, 
taking the pixel value of the background as a reference, and 
also compares the difference between the pixel value of the 
pixels adjacent to the peak P in the spatial direction X and 

2 5 the pixel value of the background with the threshold value, 
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thereby detecting the fine line region F to which the peak P 
belongs, wherein the difference between the pixel value of 
the peak P and the pixel value of the background exceeds the 
threshold value, and wherein the difference between the 
5 pixel value of the pixel adjacent in the spatial direction X 
and the pixel value of the background is equal to or below 
the threshold value. 

The monotonous increase/decrease detecting unit 2 03 
supplies to the continuousness detecting unit 2 04 monotonous 

10 increase/decrease region information indicating a region 
made up of pixels of which the pixel value monotonously 
decrease with the peak P as a reference and the sign of the 
pixel value is the same as that of the peak P, wherein the 
peak P exceeds the threshold value and wherein the pixel 

15 value of the pixel to the right side of the peak P is equal 
to or below the threshold value and the pixel value of the 
pixel to the left side of the peak P is equal to or below 
the threshold value. 

In the event of detecting a region of pixels arrayed in 

2 0 a single row in the vertical direction of the screen where 
the image of the fine line has been projected, pixels 
belonging to the region indicated by the monotonous 
increase/decrease region information are arrayed in the 
vertical direction and include pixels where the image of the 

25 fine line has been projected. That is to say, the region 
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indicated by the monotonous increase/decrease region 
information includes a region formed of pixels arrayed in a 
single row in the vertical direction of the screen where the 
image of the fine line has been projected. 
5 In this way, the peak detecting unit 2 02 and the 

monotonous increase/decrease detecting unit 203 detects a 
continuity region made up of pixels where the image of the 
fine line has been projected, employing the nature that, of 
the pixels where the image of the fine line has been 

10 projected, change in the pixel values in the spatial 
direction Y approximates Gaussian distribution. 

Of the region made up of pixels arrayed in the vertical 
direction, indicated by the monotonous increase/decrease 
region information supplied from the monotonous 

15 increase/decrease detecting unit 203, the continuousness 

detecting unit 204 detects regions including pixels adjacent 
in the horizontal direction, i.e., regions having similar 
pixel value change and duplicated in the vertical direction, 
as continuous regions, and outputs the peak information and 

20 data continuity information indicating the detected 
continuous regions. The data continuity information 
includes monotonous increase/decrease region information, 
information indicating the connection of regions, and so 
forth. 

25 Arc shapes are aligned at constant intervals in an 
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adjacent manner with the pixels where the fine line has been 
projected, so the detected continuous regions include the 
pixels where the fine line has been projected. 

The detected continuous regions include the pixels 
5 where arc shapes are aligned at constant intervals in an 
adjacent manner to which the fine line has been projected, 
so the detected continuous regions are taken as a continuity 
region, and the continuousness detecting unit 2 04 outputs 
data continuity information indicating the detected 

10 continuous regions. 

That is to say, the continuousness detecting unit 204 
uses the continuity wherein arc shapes are aligned at 
constant intervals in an adjacent manner in the data 3 
obtained by imaging the fine line, which has been generated 

15 due to the continuity of the image of the fine line in the 
actual world 1, the nature of the continuity being 
continuing in the length direction, so as to further narrow 
down the candidates of regions detected with the peak 
detecting unit 2 02 and the monotonous increase/decrease 

20 detecting unit 203. 

Fig. 39 is a diagram describing the processing for 
detecting the continuousness of monotonous increase/decrease 
regions . 

As shown in Fig. 39, in the event that a fine line 
25 region F formed of pixels aligned in a single row in the 
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vertical direction of the screen includes pixels adjacent in 
the horizontal direction, the continuousness detecting unit 
2 04 determines that there is continuousness between the two 
monotonous increase/decrease regions, and in the event that 
5 pixels adjacent in the horizontal direction are not included, 
determines that there is no continuousness between the two 
fine line regions F. For example, a fine line region F_i 
made up of pixels aligned in a single row in the vertical 
direction of the screen is determined to be continuous to a 

10 fine line region F 0 made up of pixels aligned in a single 

row in the vertical direction of the screen in the event of 
containing a pixel adjacent to a pixel of the fine line 
region F 0 in the horizontal direction. The fine line region 
F 0 made up of pixels aligned in a single row in the vertical 

15 direction of the screen is determined to be continuous to a 
fine line region Fi made up of pixels aligned in a single 
row in the vertical direction of the screen in the event of 
containing a pixel adjacent to a pixel of the fine line 
region Fi in the horizontal direction. 

2 0 In this way, regions made up of pixels aligned in a 

single row in the vertical direction of the screen where the 
image of the fine line has been projected are detected by 
the peak detecting unit 202 through the continuousness 
detecting unit 204. 

2 5 As described above, the peak detecting unit 2 02 through 
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the continuousness detecting unit 2 04 detect regions made up 
of pixels aligned in a single row in the vertical direction 
of the screen where the image of the fine line has been 
projected, and further detect regions made up of pixels 
5 aligned in a single row in the horizontal direction of the 
screen where the image of the fine line has been projected. 

Note that the order of processing is not particularly 
restricted, and may be executed in parallel, as a matter of 
course . 

10 That is to say, the peak detecting unit 202, with 

regard to of pixels aligned in a single row in the 
horizontal direction of the screen, detects as a peak a 
pixel which has a pixel value greater in comparison with the 
pixel value of the pixel situated to the left side on the 

15 screen and the pixel value of the pixel situated to the 
right side on the screen, and supplies peak information 
indicating the position of the detected peak to the 
monotonous increase/decrease detecting unit 203. The peak 
detecting unit 2 02 detects one or multiple peaks from one 

20 image, for example, one frame image. 

For example, the peak detecting unit 202 selects a 
pixel of interest from pixels in the one frame image which 
has not yet been taken as a pixel of interest, compares the 
pixel value of the pixel of interest with the pixel value of 

25 the pixel to the left side of the pixel of interest, 
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compares the pixel value of the pixel of interest with the 
pixel value of the pixel to the right side of the pixel of 
interest, detects a pixel of interest having a pixel value 
greater than the pixel value of the pixel to the left side 
5 of the pixel of interest and having a pixel value greater 
than the pixel value of the pixel to the right side of the 
pixel of interest, and takes the detected pixel of interest 
as a peak. The peak detecting unit 202 supplies peak 
information indicating the detected peak to the monotonous 

10 increase/decrease detecting unit 203. 

There are cases wherein the peak detecting unit 2 02 
does not detect a peak. 

The monotonous increase/decrease detecting unit 2 03 
detects candidates for a region made up of pixels aligned in 

15 a single row in the horizontal direction as to the peak 
detected by the peak detecting unit 202 wherein the fine 
line image has been projected, and supplies the monotonous 
increase/decrease region information indicating the detected 
region to the cont inuousness detecting unit 204 along with 

2 0 the peak information. 

More specifically, the monotonous increase/decrease 
detecting unit 203 detects regions made up of pixels having 
pixel values monotonously decreasing with the pixel value of 
the peak as a reference, as candidates of regions made up of 

2 5 pixels where the fine line image has been projected. 
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For example, the monotonous increase/decrease detecting 
unit 203 obtains, with regard to each pixel in a single row 
in the horizontal direction as to the peak, the pixel value 
of each pixel, the difference as to the pixel value of the 
5 pixel to the left side, and the difference as to the pixel 
value of the pixel to the right side. The monotonous 
increase/decrease detecting unit 2 03 then detects the region 
where the pixel value monotonously decreases by detecting 
the pixel where the sign of the difference changes. 

10 Further, the monotonous increase/decrease detecting 

unit 203 detects, from a region wherein pixel values 
monotonously decrease, a region made up of pixels having 
pixel values with the same sign as the pixel value as the 
sign of the pixel value of the peak, with reference to the 

15 sign of the pixel value of the peak, as a candidate for a 

region made up of pixels where the fine line image has been 
projected. 

For example, the monotonous increase/decrease detecting 
unit 2 03 compares the sign of the pixel value of each pixel 

20 with the sign of the pixel value of the pixel to the left 

side or with the sign of the pixel value of the pixel to the 
right side, and detects the pixel where the sign of the 
pixel value changes, thereby detecting a region made up of 
pixels having pixel values with the same sign as the peak, 

25 from the region where the pixel values monotonously decrease. 
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Thus, the monotonous increase/decrease detecting unit 
203 detects a region made up of pixels aligned in the 
horizontal direction and having pixel values with the same 
sign as the peak wherein the pixel values monotonously 
5 decrease as to the peak. 

From a fine line region made up of such a monotonous 
increase/decrease region, the monotonous increase/decrease 
detecting unit 203 obtains a fine line region longer than a 
threshold value set beforehand, i.e., a fine line region 

10 having a greater number of pixels than the threshold value. 

Further, from the fine line region thus detected, the 
monotonous increase/decrease detecting unit 203 compares the 
pixel value of the peak, the pixel value of the pixel above 
the peak, and the pixel value of the pixel below the peak, 

15 each with the threshold value, detects a fine line region to 
which belongs a peak wherein the pixel value of the peak 
exceeds the threshold value, the pixel value of the pixel 
above the peak is within the threshold, and the pixel value 
of the pixel below the peak is within the threshold, and 

20 takes the detected fine line region as a candidate for a 
region made up of pixels containing the fine line image 
component . 

Another way of saying this is that fine line regions to 
which belongs a peak wherein the pixel value of the peak is 
25 within the threshold value, or the pixel value of the pixel 
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above the peak exceeds the threshold, or the pixel value of 
the pixel below the peak exceeds the threshold, are 
determined to not contain the fine line image component, and 
are eliminated from candidates of the region made up of 
5 pixels containing the fine line image component. 

Note that the monotonous increase/decrease detecting 
unit 2 03 may be arranged to take the background pixel value 
as a reference, compare the difference between the pixel 
value of the peak and the pixel value of the background with 

10 the threshold value, and also to compare the difference 
between the pixel value of the background and the pixel 
values of the pixels adjacent to the peak in the vertical 
direction with the threshold value, and take a detected fine 
line region wherein the difference between the pixel value 

15 of the peak and the pixel value of the background exceeds 
the threshold value, and the difference between the pixel 
value of the background and the pixel value of the pixels 
adjacent in the vertical direction is within the threshold, 
as a candidate for a region made up of pixels containing the 

2 0 fine line image component. 

The monotonous increase/decrease detecting unit 2 03 
supplies to the cont inuousness detecting unit 2 04 monotonous 
increase/decrease region information indicating a region 
made up of pixels having a pixel value sign which is the 

2 5 same as the peak and monotonously decreasing pixel values as 



- 112 - 

S04P0850 

to the peak as a reference, wherein the peak exceeds the 
threshold value, and the pixel value of the pixel to the 
right side of the peak is within the threshold, and the 
pixel value of the pixel to the left side of the peak is 
5 within the threshold. 

In the event of detecting a region made up of pixels 
aligned in a single row in the horizontal direction of the 
screen wherein the image of the fine line has been projected, 
pixels belonging to the region indicated by the monotonous 

10 increase/decrease region information include pixels aligned' 
in the horizontal direction wherein the image of the fine 
line has been projected. That is to say, the region 
indicated by the monotonous increase/decrease region 
information includes a region made up of pixels aligned in a 

15 single row in the horizontal direction of the screen wherein 
the image of the fine line has been projected. 

Of the regions made up of pixels aligned in the 
horizontal direction indicated in the monotonous 
increase/decrease region information supplied from the 

20 monotonous increase/decrease detecting unit 203, the 

continuousness detecting unit 2 04 detects regions including 
pixels adjacent in the vertical direction, i.e., regions 
having similar pixel value change and which are repeated in 
the horizontal direction, as continuous regions, and outputs 

25 data continuity information indicating the peak information 
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and the detected continuous regions. The data continuity 
information includes information indicating the connection 
of the regions. 

At the pixels where the fine line has been projected, 
5 arc shapes are arrayed at constant intervals in an adjacent 
manner, so the detected continuous regions include pixels 
where the fine line has been projected. 

The detected continuous regions include pixels where 
arc shapes are arrayed at constant intervals in an adjacent 
10 manner wherein the fine line has been projected, so the 

detected continuous regions are taken as a continuity region, 
and the cont inuousness detecting unit 204 outputs data 
continuity information indicating the detected continuous 
regions . 

15 That is to say, the cont inuousness detecting unit 204 

uses the continuity which is that the arc shapes are arrayed 
at constant intervals in an adjacent manner in the data 3 
obtained by imaging the fine line, generated from the 
continuity of the image of the fine line in the actual world 

20 1 which is continuation in the length direction, so as to 
further narrow down the candidates of regions detected by 
the peak detecting unit 2 02 and the monotonous 
increase/decrease detecting unit 203. 

Thus, the data continuity detecting unit 101 is capable 

25 of detecting continuity contained in the data 3 which is the 
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input image. That is to say, the data continuity detecting 
unit 101 can detect continuity of data included in the data 
3 which has been generated by the actual world 1 image which 
is a fine line having been projected on the data 3. The 
5 data continuity detecting unit 101 detects, from the data 3, 
regions made up of pixels where the actual world 1 image 
which is a fine line has been projected. 

Fig. 4 0 is a diagram illustrating an example of other 
processing for detecting regions having continuity, where a 

10 fine line image has been projected, with the data continuity 
detecting unit 101. 

As shown in Fig. 40, the data continuity detecting unit 
101 calculates the absolute value of difference of pixel 
values for each pixel and adjacent pixels. The calculated 

15 absolute values of difference are placed corresponding to 
the pixels. For example, in a situation such as shown in 
Fig. 4 0 wherein there are pixels aligned which have 
respective pixel values of P0, PI, and P2 , the data 
continuity detecting unit 101 calculates the difference do = 

20 P0 - PI and the difference dl = PI - P2 . Further, the data 
continuity detecting unit 101 calculates the absolute values 
of the difference dO and the difference dl . 

In the event that the non-continuity component 
contained in the pixel values P0, PI, and P2 are identical, 

2 5 only values corresponding to the component of the fine line 
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are set to the difference dO and the difference dl . 

Accordingly, of the absolute values of the differences 
placed corresponding to the pixels, in the event that 
adjacent difference values are identical, the data 
5 continuity detecting unit 101 determines that the pixel 

corresponding to the absolute values of the two differences 
(the pixel between the two absolute values of difference) 
contains the component of the fine line. However, the data 
continuity detecting unit 101 does not need to detect the 
10 fine line in the event that the absolute value of difference 
is small. For example, in the event that the absolute value 
of difference is equal to or greater than a threshold value, 
the data continuity detecting unit 101 determines that the 
pixel contains a fine line component. 
15 The data continuity detecting unit 101 can also detect 

fine lines with a simple method such as this. 

Fig. 41 is a flowchart for describing continuity 
detection processing. 

In step S201, the non-continuity component extracting 
20 unit 201 extracts non-continuity component, which is 

portions other than the portion where the fine line has been 
projected, from the input image. The non- continuity 
component extracting unit 2 01 supplies non- continuity 
component information indicating the extracted non- 
25 continuity component, along with the input image, to the 
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peak detecting unit 202 and the monotonous increase/decrease 
detecting unit 203. Details of the processing for 
extracting the non-continuity component will be described 
later . 

5 In step S202, the peak detecting unit 202 eliminates 

the non- continuity component from the input image, based on 
the non-continuity component information supplied from the 
non-continuity component extracting unit 201, so as to leave 
only pixels including the continuity component in the input 

10 image. Further, in step S202, the peak detecting unit 202 
detects peaks. 

That is to say, in the event of executing processing 
with the vertical direction of the screen as a reference, of 
the pixels containing the continuity component, the peak 

15 detecting unit 2 02 compares the pixel value of each pixel 
with the pixel values of the pixels above and below, and 
detects pixels having a greater pixel value than the pixel 
value of the pixel above and the pixel value of the pixel 
below, thereby detecting a peak. Also, in step S202, in the 

20 event of executing processing with the horizontal direction 
of the screen as a reference, of the pixels containing the 
continuity component, the peak detecting unit 202 compares 
the pixel value of each pixel with the pixel values of the 
pixels to the right side and left side, and detects pixels 

25 having a greater pixel value than the pixel value of the 
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pixel to the right side and the pixel value of the pixel to 
the left side, thereby detecting a peak. 

The peak detecting unit 202 supplies the peak 
information indicating the detected peaks to the monotonous 
5 increase/decrease detecting unit 203. 

In step S2 03, the monotonous increase/decrease 
detecting unit 2 03 eliminates the non-continuity component 
from the input image, based on the non- continuity component 
information supplied from the non-continuity component 

10 extracting unit 201, so as to leave only pixels including 
the continuity component in the input image. Further, in 
step S203, the monotonous increase/decrease detecting unit 
2 03 detects the region made up of pixels having data 
continuity, by detecting monotonous increase/decrease as to 

15 the peak, based on peak information indicating the position 
of the peak, supplied from the peak detecting unit 202. 

In the event of executing processing with the vertical 
direction of the screen as a reference, the monotonous 
increase/decrease detecting unit 203 detects monotonous 

20 increase/decrease made up of one row of pixels aligned 

vertically where a single fine line image has been projected, 
based on the pixel value of the peak and the pixel values of 
the one row of pixels aligned vertically as to the peak, 
thereby detecting a region made up of pixels having data 

25 continuity. That is to say, in step S203, in the event of 
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executing processing with the vertical direction of the 
screen as a reference, the monotonous increase/decrease 
detecting unit 203 obtains, with regard to a peak and a row 
of pixels aligned vertically as to the peak, the difference 
5 between the pixel value of each pixel and the pixel value of 
a pixel above or below, thereby detecting a pixel where the 
sign of the difference changes. Also, with regard to a peak 
and a row of pixels aligned vertically as to the peak, the 
monotonous increase/decrease detecting unit 2 03 compares the 

10 sign of the pixel value of each pixel with the sign of the 
pixel value of a pixel above or below, thereby detecting a 
pixel where the sign of the pixel value changes. Further, 
the monotonous increase/decrease detecting unit 203 compares 
pixel value of the peak and the pixel values of the pixels 

15 to the right side and to the left side of the peak with a 
threshold value, and detects a region made up of pixels 
wherein the pixel value of the peak exceeds the threshold 
value, and wherein the pixel values of the pixels to the 
right side and to the left side of the peak are within the 

2 0 threshold. 

The monotonous increase/decrease detecting unit 2 03 
takes a region detected in this way as a monotonous 
increase/decrease region, and supplies monotonous 
increase/decrease region information indicating the 

25 monotonous increase/decrease region to the cont inuousness 
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detecting unit 204. 

In the event of executing processing with the 
horizontal direction of the screen as a reference, the 
monotonous increase/decrease detecting unit 2 03 detects 
5 monotonous increase/decrease made up of one row of pixels 

aligned horizontally where a single fine line image has been 
projected, based on the pixel value of the peak and the 
pixel values of the one row of pixels aligned horizontally 
as to the peak, thereby detecting a region made up of pixels 

10 having data continuity. That is to say, in step S203, in 
the event of executing processing with the horizontal 
direction of the screen as a reference, the monotonous 
increase/decrease detecting unit 203 obtains, with regard to 
a peak and a row of pixels aligned horizontally as to the 

15 peak, the difference between the pixel value of each pixel 
and the pixel value of a pixel to the right side or to the 
left side, thereby detecting a pixel where the sign of the 
difference changes. Also, with regard to a peak and a row 
of pixels aligned horizontally as to the peak, the 

20 monotonous increase/decrease detecting unit 203 compares the 
sign of the pixel value of each pixel with the sign of the 
pixel value of a pixel to the right side or to the left side, 
thereby detecting a pixel where the sign of the pixel value 
changes. Further, the monotonous increase/decrease 

2 5 detecting unit 2 03 compares pixel value of the peak and the 



- 120 - 

S04P0850 

pixel values of the pixels to the upper side and to the 
lower side of the peak with a threshold value, and detects a 
region made up of pixels wherein the pixel value of the peak 
exceeds the threshold value, and wherein the pixel values of 
5 the pixels to the upper side and to the lower side of the 
peak are within the threshold. 

The monotonous increase/decrease detecting unit 2 03 
takes a region detected in this way as a monotonous 
increase/decrease region, and supplies monotonous 

10 increase/decrease region information indicating the 

monotonous increase/decrease region to the cont inuousness 
detecting unit 204. 

In step S2 04, the monotonous increase/decrease 
detecting unit 2 03 determines whether or not processing of 

15 all pixels has ended. For example, the non-continuity 

component extracting unit 201 detects peaks for all pixels 
of a single screen (for example, frame, field, or the like) 
of the input image, and whether or not a monotonous 
increase/decrease region has been detected is determined. 

2 0 In the event that determination is made in step S2 04 

that processing of all pixels has not ended, i.e., that 
there are still pixels which have not been subjected to the 
processing of peak detection and detection of monotonous 
increase/decrease region, the flow returns to step S2 02, a 

2 5 pixel which has not yet been subjected to the processing of 
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peak detection and detection of monotonous increase/decrease 
region is selected as an object of the processing, and the 
processing of peak detection and detection of monotonous 
increase/decrease region are repeated. 
5 In the event that determination is made in step S2 04 

that processing of all pixels has ended, i.e., that peaks 
and monotonous increase/decrease regions have been detected 
with regard to all pixels, the flow proceeds to step S205, 
where the continuousness detecting unit 204 detects the 

10 continuousness of detected regions, based on the monotonous 
increase/decrease region information. For example, in the 
event that monotonous increase/decrease regions made up of 
one row of pixels aligned in the vertical direction of the 
screen, indicated by monotonous increase/decrease region 

15 information, include pixels adjacent in the horizontal 

direction, the continuousness detecting unit 204 determines 
that there is continuousness between the two monotonous 
increase/decrease regions, and in the event of not including 
pixels adjacent in the horizontal direction, determines that 

2 0 there is no continuousness between the two monotonous 

increase/decrease regions. For example, in the event that 
monotonous increase/decrease regions made up of one row of 
pixels aligned in the horizontal direction of the screen, 
indicated by monotonous increase/decrease region information, 

25 include pixels adjacent in the vertical direction, the 
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continuousness detecting unit 2 04 determines that there is 
continuousness between the two monotonous increase/decrease 
regions, and in the event of not including pixels adjacent 
in the vertical direction, determines that there is no 
5 continuousness between the two monotonous increase/decrease 
regions . 

The continuousness detecting unit 204 takes the 
detected continuous regions as continuity regions having 
data continuity, and outputs data continuity information 

10 indicating the peak position and continuity region. The 

data continuity information contains information indicating 
the connection of regions. The data continuity information 
output from the continuousness detecting unit 204 indicates 
the fine line region, which is the continuity region, made 

15 up of pixels where the actual world 1 fine line image has 
been projected. 

In step S206, a continuity direction detecting unit 205 
determines whether or not processing of all pixels has ended. 
That is to say, the continuity direction detecting unit 205 

20 determines whether or not region continuation has been 

detected with regard to all pixels of a certain frame of the 
input image . 

In the event that determination is made in step S206 
that processing of all pixels has not yet ended, i.e., that 
25 there are still pixels which have not yet been taken as the 
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object of detection of region continuation, the flow returns 
to step S205, a pixel which has not yet been subjected to 
the processing of detection of region continuity is selected 
as the pixel for processing, and the processing for 
5 detection of region continuity is repeated. 

In the event that determination is made in step S206 
that processing of all pixels has ended, i.e., that all 
pixels have been taken as the object of detection of region 
continuity, the processing ends. 

10 Thus, the continuity contained in the data 3 which is 

the input image is detected. That is to say, continuity of 
data included in the data 3 which has been generated by the 
actual world 1 image which is a fine line having been 
projected on the data 3 is detected, and a region having 

15 data continuity, which is made up of pixels on which the 

actual world 1 image which is a fine line has been projected, 
is detected from the data 3 . 

Now, the data continuity detecting unit 101 of which 
the configuration is shown in Fig. 30 can detect time- 

2 0 directional data continuity, based on the region having data 
continuity detected from the frame of the data 3 . 

For example, as shown in Fig. 42, the continuousness 
detecting unit 2 04 detects time-directional data continuity 
by connecting the edges of the region having detected data 

25 continuity in frame #n, the region having detected data 
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continuity in frame #n-l, and the region having detected 
data continuity in frame #n+l. 

The frame #n-l is a frame preceding the frame #n time- 
wise, and the frame #n+l is a frame following the frame #n 
5 time- wise. That is to say, the frame #n-l, the frame #n, 

and the frame #n+l, are displayed on the order of the frame 
#n-l, the frame #n, and the frame #n+l. 

More specifically, in Fig. 42, G denotes a movement 
vector obtained by connecting the one edge of the region 

10 having detected data continuity in frame #n, the region 
having detected data continuity in frame #n-l, and the 
region having detected data continuity in frame #n+l, and G 1 
denotes a movement vector obtained by connecting the other 
edges of the regions having detected data continuity. The 

15 movement vector G and the movement vector G' are an example 
of data continuity in the time direction. 

Further, the data continuity detecting unit 101 of 
which the configuration is shown in Fig. 3 0 can output 
information indicating the length of the region having data 

20 continuity as data continuity information. 

Fig. 43 is a block diagram illustrating the 
configuration of the non-continuity component extracting 
unit 201 which performs planar approximation of the non- 
continuity component which is the portion of the image data 

25 which does not have data continuity, and extracts the non- 
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continuity component. 

The non-continuity component extracting unit 201 of 
which the configuration is shown in Fig. 43 extracts blocks, 
which are made up of a predetermined number of pixels, from 
the input image, performs planar approximation of the blocks, 
so that the error between the block and a planar value is 
below a predetermined threshold value, thereby extracting 
the non-continuity component. 

The input image is supplied to a block extracting unit 
221, and is also output without change. 

The block extracting unit 221 extracts blocks, which 
are made up of a predetermined number of pixels, from the 
input image. For example, the block extracting unit 221 
extracts a block made up of 7 x 7 pixels, and supplies this 
to a planar approximation unit 222. For example, the block 
extracting unit 221 moves the pixel serving as the center of 
the block to be extracted in raster scan order, thereby 
sequentially extracting blocks from the input image. 

The planar approximation unit 222 approximates the 
pixel values of a pixel contained in the block on a 
predetermined plane. For example, the planar approximation 
unit 222 approximates the pixel value of a pixel contained 
in the block on a plane expressed by Expression (32) . 

z - ax+by+c 

. . . (32) 
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In Expression (32) , x represents the position of the 
pixel in one direction on the screen (the spatial direction 
X) , and y represents the position of the pixel in the other 
direction on the screen (the spatial direction Y) . z 
5 represents the application value represented by the plane, 
a represents the gradient of the spatial direction X of the 
plane, and b represents the gradient of the spatial 
direction Y of the plane. In Expression (32), c represents 
the offset of the plane (intercept) . 

10 For example, the planar approximation unit 222 obtains 

the gradient a, gradient b, and offset c, by regression 
processing, thereby approximating the pixel values of the 
pixels contained in the block on a plane expressed by 
Expression (32) . The planar approximation unit 222 obtains 

15 the gradient a, gradient b, and offset c, by regression 

processing including rejection, thereby approximating the 
pixel values of the pixels contained in the block on a plane 
expressed by Expression (32) . 

For example, the planar approximation unit 222 obtains 

2 0 the plane expressed by Expression (32) wherein the error is 
least as to the pixel values of the pixels of the block 
using the least-square method, thereby approximating the 
pixel values of the pixels contained in the block on the 
plane . 

25 Note that while the planar approximation unit 222 has 
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been described approximating the block on the plane 
expressed by Expression (32) , this is not restricted to the 
plane expressed by Expression (32) , rather, the block may be 
approximated on a plane represented with a function with a 
5 higher degree of freedom, for example, an n-order (wherein n 
is an arbitrary integer) polynomial. 

A repetition determining unit 223 calculates the error 
between the approximation value represented by the plane 
upon which the pixel values of the block have been 

10 approximated, and the corresponding pixel values of the 

pixels of the block. Expression (33) is an expression which 
shows the error ei which is the difference between the 
approximation value represented by the plane upon which the 
pixel values of the block have been approximated, and the 

15 corresponding pixel values zi of the pixels of the block. 

ej = Zj-z = Zj- (axj+byj+c) 

. . . (33) 

In Expression (33) , z-hat (A symbol with A over z will 
be described as z-hat. The same description will be used in 

20 the present specification hereafter.) represents an 

approximation value expressed by the plane on which the 
pixel values of the block are approximated, a-hat represents 
the gradient of the spatial direction X of the plane on 
which the pixel values of the block are approximated, b-hat 

25 represents the gradient of the spatial direction Y of the 
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plane on which the pixel values of the block are 
approximated, and c-hat represents the offset (intercept) of 
the plane on which the pixel values of the block are 
approximated . 

5 The repetition determining unit 223 rejects the pixel 

regarding which the error ei between the approximation value 
and the corresponding pixel values of pixels of the block is 
greatest, shown in Expression (33). Thus, pixels where the 
fine line has been projected, i.e., pixels having continuity, 

10 are rejected. The repetition determining unit 223 supplies 
rejection information indicating the rejected pixels to the 
planar approximation unit 222. 

Further, the repetition determining unit 223 calculates 
a standard error, and in the event that the standard error 

15 is equal to or greater than threshold value which has been 
set beforehand for determining ending of approximation, and 
half or more of the pixels of the pixels of a block have not 
been rejected, the repetition determining unit 223 causes 
the planar approximation unit 222 to repeat the processing 

2 0 of planar approximation on the pixels contained in the block, 
from which the rejected pixels have been eliminated. 

Pixels having continuity are rejected, so approximating 
the pixels from which the rejected pixels have been 
eliminated on a plane means that the plane approximates the 

25 non-continuity component. 
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At the point that the standard error below the 
threshold value for determining ending of approximation, or 
half or more of the pixels of the pixels of a block have 
been rejected, the repetition determining unit 223 ends 
5 planar approximation. 

With a block made up of 5 x 5 pixels, the standard 
error e s can be calculated with, for example, Expression 
(34) . 



Here, n is the number of pixels. 

Note that the repetition determining unit 223 is not 
restricted to standard error, and may be arranged to 
calculate the sum of the square of errors for all of the 
15 pixels contained in the block, and perform the following 
processing . 

Now, at the time of planar approximation of blocks 
shifted one pixel in the raster scan direction, a pixel 
having continuity, indicated by the black circle in the 
20 diagram, i.e., a pixel containing the fine line component, 
will be rejected multiple times, as shown in Fig. 44. 

Upon completing planar approximation, the repetition 
determining unit 223 outputs information expressing the 




X (Zj-z)/(n-3) 

I {(z i -(ax i +by i +c)}/(n-3) 



10 



. . . (34) 
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plane for approximating the pixel values of the block (the 
gradient and intercept of the plane of Expression 32)) as 
non-continuity information . 

Note that an arrangement may be made wherein the 
5 repetition determining unit 223 compares the number of times 
of rejection per pixel with a preset threshold value, and 
takes a pixel which has been rejected a number of times 
equal to or greater than the threshold value as a pixel 
containing the continuity component, and output the 

10 information indicating the pixel including the continuity 

component as continuity component information. In this case, 
the peak detecting unit 202 through the continuity direction 
detecting unit 205 execute their respective processing on 
pixels containing continuity component, indicated by the 

15 continuity component information. 

The number of times of rejection, the gradient of the 
spatial direction X of the plane for approximating the pixel 
values of the pixel of the block, the gradient of the 
spatial direction Y of the plane for approximating the pixel 

20 values of the pixel of the block, approximation values 

expressed by the plane approximating the pixel values of the 
pixels of the block, and the error ei, can be used as 
features of the input image . 

Fig. 45 is a flowchart for describing the processing of 

25 extracting the non-continuity component with the non- 
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continuity component extracting unit 201 of which the 
configuration is shown in Fig. 43, corresponding to step 
S201 . 

In step S221, the block extracting unit 221 extracts a 
5 block made up of a predetermined number of pixels from the 
input image, and supplies the extracted block to the planar 
approximation unit 222. For example, the block extracting 
unit 221 selects one pixel of the pixels of the input pixel 
which have not been selected yet, and extracts a block made 

10 up of 7 x 7 pixels centered on the selected pixel. For 

example, the block extracting unit 221 can select pixels in 
raster scan order. 

In step S222, the planar approximation unit 222 
approximates the extracted block on a plane. The planar 

15 approximation unit 222 approximates the pixel values of the 
pixels of the extracted block on a plane by regression 
processing, for example. For example, the planar 
approximation unit 222 approximates the pixel values of the 
pixels of the extracted block excluding the rejected pixels 

2 0 on a plane, by regression processing. In step S223, the 
repetition determining unit 223 executes repetition 
determination. For example, repetition determination is 
performed by calculating the standard error from the pixel 
values of the pixels of the block and the planar 

2 5 approximation values, and counting the number of rejected 
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pixels . 

In step S224, the repetition determining unit 223 
determines whether or not the standard error is equal to or 
above a threshold value, and in the event that determination 
5 is made that the standard error is equal to or above the 
threshold value, the flow proceeds to step S225. 

Note that an arrangement may be made wherein the 
repetition determining unit 223 determines in step S224 
whether or not half or more of the pixels of the block have 
10 been rejected, and whether or not the standard error is 

equal to or above the threshold value, and in the event that 
determination is made that half or more of the pixels of the 
block have not been rejected, and the standard error is 
equal to or above the threshold value, the flow proceeds to 
15 step S225. 

In step S225, the repetition determining unit 223 
calculates the error between the pixel value of each pixel 
of the block and the approximated planar approximation value, 
rejects the pixel with the greatest error, and notifies the 
20 planar approximation unit 222. The procedure returns to 
step S222, and the planar approximation processing and 
repetition determination processing is repeated with regard 
to the pixels of the block excluding the rejected pixel. 
In step S225, in the event that a block which is 
25 shifted one pixel in the raster scan direction is extracted 
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in the processing in step S221, the pixel including the fine 
line component (indicated by the black circle in the 
drawing) is rejected multiple times, as shown in Fig. 44. 
In the event that determination is made in step S224 
5 that the standard error is not equal to or greater than the 
threshold value, the block has been approximated on the 
plane, so the flow proceeds to step S226. 

Note that an arrangement may be made wherein the 
repetition determining unit 223 determines in step S224 

10 whether or not half or more of the pixels of the block have 
been rejected, and whether or not the standard error is 
equal to or above the threshold value, and in the event that 
determination is made that half or more of the pixels of the 
block have been rejected, or the standard error is not equal 

15 to or above the threshold value, the flow proceeds to step 
S225 . 

In step S226, the repetition determining unit 223 
outputs the gradient and intercept of the plane for 
approximating the pixel values of the pixels of the block as 
2 0 non-continuity component information. 

In step S227, the block extracting unit 221 determines 
whether or not processing of all pixels of one screen of the 
input image has ended, and in the event that determination 
is made that there are still pixels which have not yet been 
25 taken as the object of processing, the flow returns to step 
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S221, a block is extracted from pixels not yet been 
subjected to the processing, and the above processing is 
repeated . 

In the event that determination is made in step S227 
5 that processing has ended for all pixels of one screen of 
the input image, the processing ends. 

Thus, the non-continuity component extracting unit 201 
of which the configuration is shown in Fig. 43 can extract 
the non-continuity component from the input image. The non- 
10 continuity component extracting unit 2 01 extracts the non- 
continuity component from the input image, so the peak 
detecting unit 2 02 and monotonous increase/decrease 
detecting unit 203 can obtain the difference between the 
input image and the non-continuity component extracted by 
15 the non-continuity component extracting unit 201, so as to 
execute the processing regarding the difference containing 
the continuity component . 

Note that the standard error in the event that 
rejection is performed, the standard error in the event that 
2 0 rejection is not performed, the number of times of rejection 
of a pixel, the gradient of the spatial direction X of the 
plane (a-hat in Expression (32)), the gradient of the 
spatial direction Y of the plane (b-hat in Expression (32) ) , 
the level of planar transposing (c-hat in Expression (32)), 
25 and the difference between the pixel values of the input 
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image and the approximation values represented by the plane, 
calculated in planar approximation processing, can be used 
as features . 

Fig. 46 is a flowchart for describing processing for 
5 extracting the continuity component with the non-continuity 
component extracting unit 201 of which the configuration is 
shown in Fig. 43, instead of the processing for extracting 
the non- continuity component corresponding to step S201. 
The processing of step S241 through step S24 5 is the same as 

10 the processing of step S221 through step S225, so 
description thereof will be omitted. 

In step S246, the repetition determining unit 223 
outputs the difference between the approximation value 
represented by the plane and the pixel values of the input 

15 image, as the continuity component of the input image. That 
is to say, the repetition determining unit 223 outputs the 
difference between the planar approximation values and the 
true pixel values. 

Note that the repetition determining unit 223 may be 

2 0 arranged to output the difference between the approximation 
value represented by the plane and the pixel values of the 
input image, regarding pixel values of pixels of which the 
difference is equal to or greater than a predetermined 
threshold value, as the continuity component of the input 

2 5 image. 
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The processing of step S247 is the same as the 
processing of step S227 # and accordingly description thereof 
will be omitted. 

The plane approximates the non-continuity component, so 
5 the non-continuity component extracting unit 201 can remove 
the non- continuity component from the input image by 
subtracting the approximation value represented by the plane 
for approximating pixel values, from the pixel values of 
each pixel in the input image. In this case, the peak 

10 detecting unit 2 02 through the cont inuousness detecting unit 
204 can be made to process only the continuity component of 
the input image, i.e., the values where the fine line image 
has been projected, so the processing with the peak 
detecting unit 2 02 through the continuousness detecting unit 

15 204 becomes easier. 

Fig. 47 is a flowchart for describing other processing 
for extracting the continuity component with the non- 
continuity component extracting unit 2 01 of which the 
configuration is shown in Fig. 43, instead of the processing 

2 0 for extracting the non- cont inuity component corresponding to 
step S201. The processing of step S261 through step S265 is 
the same as the processing of step S221 through step S225, 
so description thereof will be omitted. 

In step S266, the repetition determining unit 223 

25 stores the number of times of rejection for each pixel, the 



- 137 - 

S04P0850 

flow returns to step S262, and the processing is repeated. 

In step S264, in the event that determination is made 
that the standard error is not equal to or greater than the 
threshold value, the block has been approximated on the 
5 plane, so the flow proceeds to step S267, the repetition 

determining unit 223 determines whether or not processing of 
all pixels of one screen of the input image has ended, and 
in the event that determination is made that there are still 
pixels which have not yet been taken as the object of 

10 processing, the flow returns to step S261, with regard to a 
pixel which has not yet been subjected to the processing, a 
block is extracted, and the above processing is repeated. 

In the event that determination is made in step S2 67 
that processing has ended for all pixels of one screen of 

15 the input image, the flow proceeds to step S268, the 

repetition determining unit 223 selects a pixel which has 
not yet been selected, and determines whether or not the 
number of times of rejection of the selected pixel is equal 
to or greater than a threshold value. For example, the 

20 repetition determining unit 223 determines in step S268 
whether or not the number of times of rejection of the 
selected pixel is equal to or greater than a threshold value 
stored beforehand. 

In the event that determination is made in step S2 68 

25 that the number of times of rejection of the selected pixel 
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is equal to or greater than the threshold value, the 
selected pixel contains the continuity component, so the 
flow proceeds to step S269, where the repetition determining 
unit 223 outputs the pixel value of the selected pixel (the 
5 pixel value in the input image) as the continuity component 
of the input image, and the flow proceeds to step S270. 

In the event that determination is made in step S268 
that the number of times of rejection of the selected pixel 
is not equal to or greater than the threshold value, the 

10 selected pixel does not contain the continuity component, so 
the processing in step S269 is skipped, and the procedure 
proceeds to step S270. That is to say, the pixel value of a 
pixel regarding which determination has been made that the 
number of times of rejection is not equal to or greater than 

15 the threshold value is not output. 

Note that an arrangement may be made wherein the 
repetition determining unit 223 outputs a pixel value set to 
0 for pixels regarding which determination has been made 
that the number of times of rejection is not equal to or 

20 greater than the threshold value. 

In step S270, the repetition determining unit 223 
determines whether or not processing of all pixels of one 
screen of the input image has ended to determine whether or 
not the number of times of rejection is equal to or greater 

2 5 than the threshold value, and in the event that 
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determination is made that processing has not ended for all 
pixels, this means that there are still pixels which have 
not yet been taken as the object of processing, so the flow 
returns to step S268, a pixel which has not yet been 
5 subjected to the processing is selected, and the above 
processing is repeated. 

In the event that determination is made in step S270 
that processing has ended for all pixels of one screen of 
the input image, the processing ends. 

10 Thus, of the pixels of the input image, the non- ' 

continuity component extracting unit 2 01 can output the 
pixel values of pixels containing the continuity component, 
as continuity component information. That is to say, of the 
pixels of the input image, the non-continuity component 

15 extracting unit 2 01 can output the pixel values of pixels 
containing the component of the fine line image. 

Fig. 48 is a flowchart for describing yet other 
processing for extracting the continuity component with the 
non-continuity component extracting unit 201 of which the 

20 configuration is shown in Fig. 43, instead of the processing 
for extracting the non- continuity component corresponding to 
step S201. The processing of step S281 through step S288 is 
the same as the processing of step S261 through step S268, 
so description thereof will be omitted. 

25 In step S289, the repetition determining unit 223 
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outputs the difference between the approximation value 
represented by the plane, and the pixel value of a selected 
pixel, as the continuity component of the input image. That 
is to say, the repetition determining unit 223 outputs an 
5 image wherein the non- continuity component has been removed 
from the input image, as the continuity information. 

The processing of step S290 is the same as the 
processing of step S270, and accordingly description thereof 
will be omitted. 

10 Thus, the non- continuity component extracting unit 2 01 

can output an image wherein the non-continuity component has 
been removed from the input image as the continuity 
information. 

As described above, in a case wherein real world light 
15 signals are projected, a non- continuous portion of pixel 

values of multiple pixels of first image data wherein a part 
of the continuity of the real world light signals has been 
lost is detected, data continuity is detected from the 
detected non- continuous portions, a model (function) is 
20 generated for approximating the light signals by estimating 
the continuity of the real world light signals based on the 
detected data continuity, and second image data is generated 
based on the generated function, processing results which 
are more accurate and have higher precision as to the event 
25 in the real world can be obtained. 
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Fig. 49 is a block diagram illustrating another 
configuration of the data continuity detecting unit 101. 

With the data continuity detecting unit 101 of which 
the configuration is shown in Fig. 49, change in the pixel 
5 value of the pixel of interest which is a pixel of interest 
in the spatial direction of the input image, i.e. activity 
in the spatial direction of the input image, is detected, 
multiple sets of pixels made up of a predetermined number of 
pixels in one row in the vertical direction or one row in 

10 the horizontal direction are extracted for each angle based 
on the pixel of interest and a reference axis according to 
the detected activity, the correlation of the extracted 
pixel sets is detected, and the angle of data continuity 
based on the reference axis in the input image is detected 

15 based on the correlation. 

The angle of data continuity means an angle assumed by 
the reference axis, and the direction of a predetermined 
dimension where continuity repeatedly appear in the data 3. 
Continuity repeatedly appearing means a case wherein, for 

2 0 example, the change in value as to the change in position in 
the data 3, i.e., the cross-sectional shape, is the same, 
and so forth. 

The reference axis may be, for example, an axis 
indicating the spatial direction X (the horizontal direction 

25 of the screen) , an axis indicating the spatial direction Y 
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(the vertical direction of the screen) , and so forth. 

The input image is supplied to an activity detecting 
unit 401 and data selecting unit 402. 

The activity detecting unit 401 detects change in the 
5 pixel values as to the spatial direction of the input image, 
i.e., activity in the spatial direction, and supplies the 
activity information which indicates the detected results to 
the data selecting unit 402 and a continuity direction 
derivation unit 404. 

10 For example, the activity detecting unit 401 detects 

the change of a pixel value as to the horizontal direction 
of the screen, and the change of a pixel value as to the 
vertical direction of the screen, and compares the detected 
change of the pixel value in the horizontal direction and 

15 the change of the pixel value in the vertical direction, 

thereby detecting whether the change of the pixel value in 
the horizontal direction is greater as compared with the 
change of the pixel value in the vertical direction, or 
whether the change of the pixel value in the vertical 

20 direction is greater as compared with the change of the 
pixel value in the horizontal direction. 

The activity detecting unit 401 supplies to the data 
selecting unit 402 and the continuity direction derivation 
unit 404 activity information, which is the detection 

25 results, indicating that the change of the pixel value in 
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the horizontal direction is greater as compared with the 
change of the pixel value in the vertical direction, or 
indicating that the change of the pixel value in the 
vertical direction is greater as compared with the change of 
5 the pixel value in the horizontal direction. 

In the event that the change of the pixel value in the 
horizontal direction is greater as compared with the change 
of the pixel value in the vertical direction, arc shapes 
(half-disc shapes) or pawl shapes are formed on one row of 

10 pixels in the vertical direction, as indicated by Fig. 50 
for example, and the arc shapes or pawl shapes are formed 
repetitively more in the vertical direction. That is to say, 
in the event that the change of the pixel value in the 
horizontal direction is greater as compared with the change 

15 of the pixel value in the vertical direction, with the 
reference axis as the axis representing the spatial 
direction X, the angle of the data continuity based on the 
reference axis in the input image is a value of any from 45 
degrees to 90 degrees. 

2 0 In the event that the change of the pixel value in the 

vertical direction is greater as compared with the change of 
the pixel value in the horizontal direction, arc shapes or 
pawl shapes are formed on one row of pixels in the vertical 
direction, for example, and the arc shapes or pawl shapes 

25 are formed repetitively more in the horizontal direction. 
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That is to say, in the event that the change of the pixel 
value in the vertical direction is greater as compared with 
the change of the pixel value in the horizontal direction, 
with the reference axis as the axis representing the spatial 
5 direction X, the angle of the data continuity based on the 
reference axis in the input image is a value of any from 0 
degrees to 45 degrees . 

For example, the activity detecting unit 401 extracts 
from the input image a block made up of the 9 pixels, 3x3 

10 centered on the pixel of interest, as shown in Fig. 51. The 
activity detecting unit 401 calculates the sum of 
differences of the pixels values regarding the pixels 
vertically adjacent, and the sum of differences of the 
pixels values regarding the pixels horizontally adjacent. 

15 The sum of differences h diff of the pixels values regarding 
the pixels horizontally adjacent can be obtained with 
Expression (35) . 



pixels values regarding the pixels vertically adjacent can 
be obtained with Expression (3 6) . 



h d iff = 1 (P| + i.j-p u ) 



. . . (35) 



20 



In the same way, the sum of differences v d ±ff of the 



Vdiff 



= 1 (Pi 



. j+i 
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In Expression (35) and Expression (36) , P represents 
the pixel value, i represents the position of the pixel in 
the horizontal direction, and j represents the position of 
the pixel in the vertical direction. 

An arrangement may be made wherein the activity 
detecting unit 401 compares the calculated sum of 
differences h diff of the pixels values regarding the pixels 
horizontally adjacent with the sum of differences v diff of 
the pixels values regarding the pixels vertically adjacent, 
so as to determine the range of the angle of the data 
continuity based on the reference axis in the input image. 
That is to say, in this case, the activity detecting unit 
4 01 determines whether a shape indicated by change in the 
pixel value as to the position in the spatial direction is 
formed repeatedly in the horizontal direction, or formed 
repeatedly in the vertical direction. 

For example, change in pixel values in the horizontal 
direction with regard to an arc formed on pixels in one 
horizontal row is greater than the change of pixel values in 
the vertical direction, change in pixel values in the 
vertical direction with regard to an arc formed on pixels in 
one horizontal row is greater than the change of pixel 
values in the horizontal direction, and it can be said that 
the direction of data continuity, i.e., the change in the 
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direction of the predetermined dimension of a constant 
feature which the input image that is the data 3 has is 
smaller in comparison with the change in the orthogonal 
direction too the data continuity. In other words, the 
5 difference of the direction orthogonal to the direction of 
data continuity (hereafter also referred to as non- 
continuity direction) is greater as compared to the 
difference in the direction of data continuity. 

For example, as shown in Fig. 52, the activity 

10 detecting unit 4 01 compares the calculated sum of 

differences h dif f of the pixels values regarding the pixels 
horizontally adjacent with the sum of differences v d i ff of 
the pixels values regarding the pixels vertically adjacent, 
and in the event that the sum of differences h dif f of the 

15 pixels values regarding the pixels horizontally adjacent is 
greater, determines that the angle of the data continuity 
based on the reference axis is a value of any from 45 
degrees to 135 degrees, and in the event that the sum of 
differences v diff of the pixels values regarding the pixels 

20 vertically adjacent is greater, determines that the angle of 
the data continuity based on the reference axis is a value 
of any from 0 degrees to 4 5 degrees, or a value of any from 
135 degrees to 180 degrees. 

For example, the activity detecting unit 401 supplies 

25 activity information indicating the determination results to 
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the data selecting unit 402 and the continuity direction 
derivation unit 404. 

Note that the activity detecting unit 401 can detect 
activity by extracting blocks of arbitrary sizes, such as a 
5 block made up of 25 pixels of 5 x 5 , a block made up of 49 
pixels of 7 x 7, and so forth. 

The data selecting unit 402 sequentially selects pixels 
of interest from the pixels of the input image, and extracts 
multiple sets of pixels made up of a predetermined number of 

10 pixels in one row in the vertical direction or one row in 
the horizontal direction for each angle based on the pixel 
of interest and the reference axis, based on the activity 
information supplied from the activity detecting unit 401. 

For example, in the event that the activity information 

15 indicates that the change in pixel values in the horizontal 
direction is greater in comparison with the change in pixel 
values in the vertical direction, this means that the data 
continuity angle is a value of any from 45 degrees to 135 
degrees, so the data selecting unit 402 extracts multiple 

20 sets of pixels made up of a predetermined number of pixels 

in one row in the vertical direction, for each predetermined 
angle in the range of 4 5 degrees to 13 5 degrees, based on 
the pixel of interest and the reference axis. 

In the event that the activity information indicates 

25 that the change in pixel values in the vertical direction is 
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greater in comparison with the change in pixel values in the 
horizontal direction, this means that the data continuity 
angle is a value of any from 0 degrees to 45 degrees or from 
135 degrees to 180 degrees, so the data selecting unit 402 
5 extracts multiple sets of pixels made up of a predetermined 
number of pixels in one row in the horizontal direction, for 
each predetermined angle in the range of 0 degrees to 45 
degrees or 135 degrees to 180 degrees, based on the pixel of 
interest and the reference axis. 

10 Also, for example, in the event that the activity 

information indicates that the angle of data continuity is a 
value of any from 45 degrees to 135 degrees, the data 
selecting unit 402 extracts multiple sets of pixels made up 
of a predetermined number of pixels in one row in the 

15 vertical direction, for each predetermined angle in the 

range of 45 degrees to 13 5 degrees, based on the pixel of 
interest and the reference axis. 

In the event that the activity information indicates 
that the angle of data continuity is a value of any from 0 

20 degrees to 45 degrees or from 135 degrees to 180 degrees, 

the data selecting unit 402 extracts multiple sets of pixels 
made up of a predetermined number of pixels in one row in 
the horizontal direction, for each predetermined angle in 
the range of 0 degrees to 45 degrees or 135 degrees to 180 

25 degrees, based on the pixel of interest and the reference 
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axis . 

The data selecting unit 402 supplies the multiple sets 
made up of the extracted pixels to an error estimating unit 
403 . 

5 The error estimating unit 403 detects correlation of 

pixel sets for each angle with regard to the multiple sets 
of extracted pixels. 

For example, with regard to the multiple sets of pixels 
made up of a predetermined number of pixels in one row in 

10 the- vertical direction corresponding to one angle, the error 
estimating unit 403 detects the correlation of the pixels 
values of the pixels at corresponding positions of the pixel 
sets. With regard to the multiple sets of pixels made up of 
a predetermined number of pixels in one row in the 

15 horizontal direction corresponding to one angle, the error 
estimating unit 403 detects the correlation of the pixels 
values of the pixels at corresponding positions of the sets. 

The error estimating unit 4 03 supplies correlation 
information indicating the detected correlation to the 

20 continuity direction derivation unit 404. The error 

estimating unit 403 calculates the sum of the pixel values 
of pixels of a set including the pixel of interest supplied 
from the data selecting unit 402 as values indicating 
correlation, and the absolute value of difference of the 

25 pixel values of the pixels at corresponding positions in 
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other sets, and supplies the sum of absolute value of 
difference to the continuity direction derivation unit 404 
as correlation information. 

Based on the correlation information supplied from the 
5 error estimating unit 403, the continuity direction 

derivation unit 4 04 detects the data continuity angle based 
on the reference axis in the input image, corresponding to 
the lost continuity of the light signals of the actual world 
1, and outputs data continuity information indicating an 

10 angle. For example, based on the correlation information 

supplied from the error estimating unit 403, the continuity 
direction derivation unit 404 detects an angle corresponding 
to the pixel set with the greatest correlation as the data 
continuity angle, and outputs data continuity information 

15 indicating the angle corresponding to the pixel set with the 
greatest correlation that has been detected. 

The following description will be made regarding 
detection of data continuity angle in the range of 0 degrees 
through 90 degrees (the so-called first quadrant) . 

20 Fig. 53 is a block diagram illustrating a more detailed 

configuration of the data continuity detecting unit 101 
shown in Fig. 49. 

The data selecting unit 402 includes pixel selecting 
unit 411-1 through pixel selecting unit 411-L. The error 

25 estimating unit 403 includes estimated error calculating 
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unit 412-1 through estimated error calculating unit 412-L. 
The continuity direction derivation unit 404 includes a 
smallest error angle selecting unit 413. 

First, description will be made regarding the 
processing of the pixel selecting unit 411-1 through pixel 
selecting unit 411 -L in the event that the data continuity 
angle indicated by the activity information is a value of 
any from 45 degrees to 135 degrees. 

The pixel selecting unit 411-1 through pixel selecting 
unit 411-L set straight lines of mutually differing 
predetermined angles which pass through the pixel of 
interest, with the axis indicating the spatial direction X 
as the reference axis. The pixel selecting unit 411-1 
through pixel selecting unit 411-L select, of the pixels 
belonging to a vertical row of pixels to which the pixel of 
interest belongs, a predetermined number of pixels above the 
pixel of interest, and predetermined number of pixels below 
the pixel of interest, and the pixel of interest, as a set. 

For example, as shown in Fig. 54, the pixel selecting 
unit 411-1 through pixel selecting unit 411-L select 9 
pixels centered on the pixel of interest, as a set of pixels, 
from the pixels belonging to a vertical row of pixels to 
which the pixel of interest belongs. 

In Fig. 54, one grid- shaped square (one grid) 
represents one pixel. In Fig. 54, the circle shown at the 
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center represents the pixel of interest. 

The pixel selecting unit 411-1 through pixel selecting 
unit 411-L select, from pixels belonging to a vertical row 
of pixels to the left of the vertical row of pixels to which 
5 the pixel of interest belongs, a pixel at the position 

closest to the straight line set for each. In Fig. 54, the 
circle to the lower left of the pixel of interest represents 
an example of a selected pixel. The pixel selecting unit 
411-1 through pixel selecting unit 411-L then select, from 

10 the pixels belonging to the vertical row of pixels to the 
left of the vertical row of pixels to which the pixel of 
interest belongs, a predetermined number of pixels above the 
selected pixel, a predetermined number of pixels below the 
selected pixel, and the selected pixel, as a set of pixels. 

15 For example, as shown in Fig. 54, the pixel selecting 

unit 411-1 through pixel selecting unit 411-L select 9 
pixels centered on the pixel at the position closest to the 
straight line, from the pixels belonging to the vertical row 
of pixels to the left of the vertical row of pixels to which 

20 the pixel of interest belongs, as a set of pixels. 

The pixel selecting unit 411-1 through pixel selecting 
unit 411-L select, from pixels belonging to a vertical row 
of pixels second left from the vertical row of pixels to 
which the pixel of interest belongs, a pixel at the position 

25 closest to the straight line set for each. In Fig. 54, the 
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circle to the far left represents an example of the selected 
pixel. The pixel selecting unit 411-1 through pixel 
selecting unit 411-L then select, as a set of pixels, from 
the pixels belonging to the vertical row of pixels second 
5 left from the vertical row of pixels to which the pixel of 
interest belongs, a predetermined number of pixels above the 
selected pixel, a predetermined number of pixels below the 
selected pixel, and the selected pixel. 

For example, as shown in Fig. 54, the pixel selecting 

10 unit 411-1 through pixel selecting unit 411-L select 9 

pixels centered on the pixel at the position closest to the 
straight line, from the pixels belonging to the vertical row 
of pixels second left from the vertical row of pixels to 
which the pixel of interest belongs, as a set of pixels. 

15 The pixel selecting unit 411-1 through pixel selecting 

unit 411-L select, from pixels belonging to a vertical row 
of pixels to the right of the vertical row of pixels to 
which the pixel of interest belongs, a pixel at the position 
closest to the straight line set for each. In Fig. 54, the 

20 circle to the upper right of the pixel of interest 

represents an example of a selected pixel. The pixel 
selecting unit 411-1 through pixel selecting unit 411-L then 
select, from the pixels belonging to the vertical row of 
pixels to the right of the vertical row of pixels to which 

25 the pixel of interest belongs, a predetermined number of 
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pixels above the selected pixel, a predetermined number of 
pixels below the selected pixel, and the selected pixel, as 
a set of pixels. 

For example, as shown in Fig. 54, the pixel selecting 
5 unit 411-1 through pixel selecting unit 411-L select 9 

pixels centered on the pixel at the position closest to the 
straight line, from the pixels belonging to the vertical row 
of pixels to the right of the vertical row of pixels to 
which the pixel of interest belongs, as a set of pixels. 

10 The pixel selecting unit 411-1 through pixel selecting 

unit 411-L select, from pixels belonging to a vertical row 
of pixels second right from the vertical row of pixels to 
which the pixel of interest belongs, a pixel at the position 
closest to the straight line set for each. In Fig. 54, the 

15 circle to the far right represents an example of the 

selected pixel. The pixel selecting unit 411-1 through 
pixel selecting unit 411-L then select, from the pixels 
belonging to the vertical row of pixels second right from 
the vertical row of pixels to which the pixel of interest 

2 0 belongs, a predetermined number of pixels above the selected 
pixel, a predetermined number of pixels below the selected 
pixel, and the selected pixel, as a set of pixels. 

For example, as shown in Fig. 54, the pixel selecting 
unit 411-1 through pixel selecting unit 411-L select 9 

25 pixels centered on the pixel at the position closest to the 



- 155 - 

S04P0850 

straight line, from the pixels belonging to the vertical row 
of pixels second right from the vertical row of pixels to 
which the pixel of interest belongs, as a set of pixels. 
Thus, the pixel selecting unit 411-1 through pixel 
5 selecting unit 411-L each select five sets of pixels. 

The pixel selecting unit 411-1 through pixel selecting 
unit 411-L select pixel sets for (lines set to) mutually 
different angles. For example, the pixel selecting unit 
411-1 selects sets of pixels regarding 45 degrees, the pixel 

10 selecting unit 411-2 selects sets of pixels regarding 47.5 
degrees, and the pixel selecting unit 411-3 selects sets of 
pixels regarding 50 degrees. The pixel selecting unit 411-1 
through pixel selecting unit 411-L select sets of pixels 
regarding angles every 2.5 degrees, from 52.5 degrees 

15 through 135 degrees. 

Note that the number of pixel sets may be an optional 
number, such as 3 or 7, for example, and does not restrict 
the present invention. Also, the number of pixels selected 
as one set may be an optional number, such as 5 or 13, for 

20 example, and does not restrict the present invention. 

Note that the pixel selecting unit 411-1 through pixel 
selecting unit 411-L may be arranged to select pixel sets 
from pixels within a predetermined range in the vertical 
direction. For example, the pixel selecting unit 411-1 

25 through pixel selecting unit 411-L can select pixel sets 
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from 121 pixels in the vertical direction (60 pixels upward 
from the pixel of interest, and 60 pixels downward) . In 
this case, the data continuity detecting unit 101 can detect 
the angle of data continuity up to 88.09 degrees as to the 
5 axis representing the spatial direction X. 

The pixel selecting unit 411-1 supplies the selected 
set of pixels to the estimated error calculating unit 412-1, 
and the pixel selecting unit 411-2 supplies the selected set 
of pixels to the estimated error calculating unit 412-2. In 

10 the same way, each pixel selecting unit 411-3 through pixel 
selecting unit 411-L supplies the selected set of pixels to 
each estimated error calculating unit 412-3 through 
estimated error calculating unit 412 -L. 

The estimated error calculating unit 412-1 through 

15 estimated error calculating unit 412 -L detect the 

correlation of the pixels values of the pixels at positions 
in the multiple sets, supplied from each of the pixel 
selecting unit 411-1 through pixel selecting unit 411-L. 
For example, the estimated error calculating unit 412-1 

20 through estimated error calculating unit 412 -L calculates, 
as a value indicating the correlation, the sum of absolute 
values of difference between the pixel values of the pixels 
of the set containing the pixel of interest, and the pixel 
values of the pixels at corresponding positions in other 

25 sets, supplied from one of the pixel selecting unit 411-1 
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through pixel selecting unit 411 -L. 

More specifically, based on the pixel values of the 
pixels of the set containing the pixel of interest and the 
pixel values of the pixels of the set made up of pixels 
5 belonging to one vertical row of pixels to the left side of 
the pixel of interest supplied from one of the pixel 
selecting unit 411-1 through pixel selecting unit 411-L, the 
estimated error calculating unit 412-1 through estimated 
error calculating unit 412-L calculates the difference of 

10 the pixel values of the topmost pixel, then calculates the 
difference of the pixel values of the second pixel from the 
top, and so on to calculate the absolute values of 
difference of the pixel values in order from the top pixel, 
and further calculates the sum of absolute values of the 

15 calculated differences. Based on the pixel values of the 
pixels of the set containing the pixel of interest and the 
pixel values of the pixels of the set made up of pixels 
belonging to one vertical row of pixels two to the left from 
the pixel of interest supplied from one of the pixel 

20 selecting unit 411-1 through pixel selecting unit 411-L, the 
estimated error calculating unit 412-1 through estimated 
error calculating unit 412-L calculates the absolute values 
of difference of the pixel values in order from the top 
pixel, and calculates the sum of absolute values of the 

25 calculated differences. 
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Then, based on the pixel values of the pixels of the 
set containing the pixel of interest and the pixel values of 
the pixels of the set made up of pixels belonging to one 
vertical row of pixels to the right side of the pixel of 
interest supplied from one of the pixel selecting unit 411-1 
through pixel selecting unit 411-L, the estimated error 
calculating unit 412-1 through estimated error calculating 
unit 412 -L calculates the difference of the pixel values of 
the topmost pixel, then calculates the difference of the 
pixel values of the second pixel from the top, and so on to 
calculate the absolute values of difference of the pixel 
values in order from the top pixel, and further calculates 
the sum of absolute values of the calculated differences. 
Based on the pixel values of the pixels of the set 
containing the pixel of interest and the pixel values of the 
pixels of the set made up of pixels belonging to one 
vertical row of pixels two to the right from the pixel of 
interest supplied from one of the pixel selecting unit 411-1 
through pixel selecting unit 411-L, the estimated error 
calculating unit 412-1 through estimated error calculating 
unit 412 -L calculates the absolute values of difference of 
the pixel values in order from the top pixel, and calculates 
the sum of absolute values of the calculated differences. 

The estimated error calculating unit 412-1 through 
estimated error calculating unit 412 -L add all of the sums 
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of absolute values of difference of the pixel values thus 
calculated, thereby calculating the aggregate of absolute 
values of difference of the pixel values. 

The estimated error calculating unit 412-1 through 
5 estimated error calculating unit 412 -L supply information 
indicating the detected correlation to the smallest error 
angle selecting unit 413. For example, the estimated error 
calculating unit 412-1 through estimated error calculating 
unit 412 -L supply the aggregate of absolute values of 

10 difference of the pixel values calculated, to the smallest 
error angle selecting unit 413. 

Note that the estimated error calculating unit 412-1 
through estimated error calculating unit 412-L are not 
restricted to the sum of absolute values of difference of 

15 pixel values, and can also calculate other values as 

correlation values as well, such as the sum of squared 
differences of pixel values, or correlation coefficients 
based on pixel values, and so forth. 

The smallest error angle selecting unit 413 detects the 

20 data continuity angle based on the reference axis in the 

input image which corresponds to the continuity of the image 
which is the lost actual world 1 light signals, based on the 
correlation detected by the estimated error calculating unit 
412-1 through estimated error calculating unit 412-L with 

25 regard to mutually different angles. That is to say, based 
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on the correlation detected by the estimated error 
calculating unit 412-1 through estimated error calculating 
unit 412 -L with regard to mutually different angles, the 
smallest error angle selecting unit 413 selects the greatest 
5 correlation, and takes the angle regarding which the 

selected correlation was detected as the data continuity 
angle based on the reference axis, thereby detecting the 
data continuity angle based on the reference axis in the 
input image . 

10 For example, of the aggregates of absolute values of 

difference of the pixel values supplied from the estimated 
error calculating unit 412-1 through estimated error 
calculating unit 412 -L, the smallest error angle selecting 
unit 413 selects the smallest aggregate. With regard to the 

15 pixel set of which the selected aggregate was calculated, 

the smallest error angle selecting unit 413 makes reference 
to a pixel belonging to the one vertical row of pixels two 
to the left from the pixel of interest and at the closest 
pixel position to the straight line, and to a pixel 

20 belonging to the one vertical row of pixels two to the right 
from the pixel of interest and at the closest pixel position 
to the straight line. 

As shown in Fig. 54, the smallest error angle selecting 
unit 413 obtains the distance S in the vertical direction of 

25 the position of the pixels to reference from the position of 
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the pixel of interest. As shown in Fig. 55, the smallest 
error angle selecting unit 413 detects the angle 9 of data 
continuity based on the axis indicating the spatial 
direction X which is the reference axis in the input image 
5 which is image data, that corresponds to the lost actual 
world 1 light signals continuity, from Expression (37) . 

0 = tan"" 1 y 

. . . (37) 

Next, description will be made regarding the processing 
10 of the pixel selecting unit 411-1 through pixel selecting 
unit 411-L in the event that the data continuity angle 
indicated by the activity information is a value of any from 
0 degrees to 45 degrees and 135 degrees to 180 degrees. 

The pixel selecting unit 411-1 through pixel selecting 
15 unit 411-L set straight lines of predetermined angles which 
pass through the pixel of interest, with the axis indicating 
the spatial direction X as the reference axis, and select, 
of the pixels belonging to a horizontal row of pixels to 
which the pixel of interest belongs, a predetermined number 
20 of pixels to the left of the pixel of interest, and 

predetermined number of pixels to the right of the pixel of 
interest, and the pixel of interest, as a pixel set. 

The pixel selecting unit 411-1 through pixel selecting 
unit 411-L select, from pixels belonging to a horizontal row 
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of pixels above the horizontal row of pixels to which the 
pixel of interest belongs, a pixel at the position closest 
to the straight line set for each. The pixel selecting unit 
411-1 through pixel selecting unit 411-L then select, from 
5 the pixels belonging to the horizontal row of pixels above 
the horizontal row of pixels to which the pixel of interest 
belongs, a predetermined number of pixels to the left of the 
selected pixel, a predetermined number of pixels to the 
right of the selected pixel, and the selected pixel, as a 

10 pixel set. 

The pixel selecting unit 411-1 through pixel selecting 
unit 411-L select, from pixels belonging to a horizontal row 
of pixels two above the horizontal row of pixels to which 
the pixel of interest belongs, a pixel at the position 

15 closest to the straight line set for each. The pixel 

selecting unit 411-1 through pixel selecting unit 411-L then 
select, from the pixels belonging to the horizontal row of 
pixels two above the horizontal row of pixels to which the 
pixel of interest belongs, a predetermined number of pixels 

20 to the left of the selected pixel, a predetermined number of 
pixels to the right of the selected pixel, and the selected 
pixel, as a pixel set. 

The pixel selecting unit 411-1 through pixel selecting 
unit 411-L select, from pixels belonging to a horizontal row 

25 of pixels below the horizontal row of pixels to which the 
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pixel of interest belongs, a pixel at the position closest 
to the straight line set for each. The pixel selecting unit 
411-1 through pixel selecting unit 411-L then select, from 
the pixels belonging to the horizontal row of pixels below 
5 the horizontal row of pixels to which the pixel of interest 
belongs, a predetermined number of pixels to the left of the 
selected pixel, a predetermined number of pixels to the 
right of the selected pixel, and the selected pixel, as a 
pixel set . 

10 The pixel selecting unit 411-1 through pixel selecting 

unit 411-L select, from pixels belonging to a horizontal row 
of pixels two below the horizontal row of pixels to which 
the pixel of interest belongs, a pixel at the position 
closest to the straight line set for each. The pixel 

15 selecting unit 411-1 through pixel selecting unit 411-L then 
select, from the pixels belonging to the horizontal row of 
pixels two below the horizontal row of pixels to which the 
pixel of interest belongs, a predetermined number of pixels 
to the left of the selected pixel, a predetermined number of 

20 pixels to the right of the selected pixel, and the selected 
pixel, as a pixel set. 

Thus, the pixel selecting unit 411-1 through pixel 
selecting unit 411-L each select five sets of pixels. 

The pixel selecting unit 411-1 through pixel selecting 

25 unit 411-L select pixel sets for mutually different angles. 
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For example, the pixel selecting unit 411-1 selects sets of 
pixels regarding 0 degrees, the pixel selecting unit 411-2 
selects sets of pixels regarding 2.5 degrees, and the pixel 
selecting unit 411-3 selects sets of pixels regarding 5 
5 degrees. The pixel selecting unit 411-1 through pixel 

selecting unit 411-L select sets of pixels regarding angles 
every 2.5 degrees, from 7.5 degrees through 45 degrees and 
from 135 degrees through 180 degrees. 

The pixel selecting unit 411-1 supplies the selected 

10 set of pixels to the estimated error calculating unit 412-1, 
and the pixel selecting unit 411-2 supplies the selected set 
of pixels to the estimated error calculating unit 412-2. In 
the same way, each pixel selecting unit 411-3 through pixel 
selecting unit 411-L supplies the selected set of pixels to 

15 each estimated error calculating unit 412-3 through 
estimated error calculating unit 412 -L. 

The estimated error calculating unit 412-1 through 
estimated error calculating unit 412 -L detect the 
correlation of the pixels values of the pixels at positions 

20 in the multiple sets, supplied from each of the pixel 

selecting unit 411-1 through pixel selecting unit 411-L. 
The estimated error calculating unit 412-1 through estimated 
error calculating unit 412 -L supply information indicating 
the detected correlation to the smallest error angle 

25 selecting unit 413. 



/ 
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The smallest error angle selecting unit 413 detects the 
data continuity angle based on the reference axis in the 
input image which corresponds to the continuity of the image 
which is the lost actual world 1 light signals, based on the 
5 correlation detected by the estimated error calculating unit 
412-1 through estimated error calculating unit 412-L. 

Next, data continuity detection processing with the 
data continuity detecting unit 101 of which the 
configuration is shown in Fig. 49, corresponding to the 
10 processing in step S101, will be described with reference to 
the flowchart in Fig. 56. 

In step S401, the activity detecting unit 401 and the 
data selecting unit 402 select the pixel of interest which 
is a pixel of interest from the input image. The activity 
15 detecting unit 401 and the data selecting unit 402 select 
the same pixel of interest. For example, the activity 
detecting unit 401 and the data selecting unit 402 select 
the pixel of interest from the input image in raster scan 
order . 

20 In step S402, the activity detecting unit 401 detects 

activity with regard to the pixel of interest. For example, 
the activity detecting unit 401 detects activity based on 
the difference of pixel values of pixels aligned in the 
vertical direction of a block made up of a predetermined 

25 number of pixels centered on the pixel of interest, and the 
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difference of pixel values of pixels aligned in the 
horizontal direction . 

The activity detecting unit 401 detects activity in the 
spatial direction as to the pixel of interest, and supplies 
5 activity information indicating the detected results to the 
data selecting unit 402 and the continuity direction 
derivation unit 404. 

In step S403, the data selecting unit 402 selects, from 
a row of pixels including the pixel of interest, a 

10 predetermined number of pixels centered on the pixel of 

interest, as a pixel set. For example, the data selecting 
unit 4 02 selects a predetermined number of pixels above or 
to the left of the pixel of interest, and a predetermined 
number of pixels below or to the right of the pixel of 

15 interest, which are pixels belonging to a vertical or 

horizontal row of pixels to which the pixel of interest 
belongs, and also the pixel of interest, as a pixel set. 

In step S404, the data selecting unit 402 selects, as a 
pixel set, a predetermined number of pixels each from a 

2 0 predetermined number of pixel rows for each angle in a 

predetermined range based on the activity detected by the 
processing in step S402. For example, the data selecting 
unit 402 sets straight lines with angles of a predetermined 
range which pass through the pixel of interest, with the 

25 axis indicating the spatial direction X as the reference 
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axis, selects a pixel which is one or two rows away from the 
pixel of interest in the horizontal direction or vertical 
direction and which is closest to the straight line, and 
selects a predetermined number of pixels above or to the 
5 left of the selected pixel, and a predetermined number of 
pixels below or to the right of the selected pixel, and the 
selected pixel closest to the line, as a pixel set. The 
data selecting unit 402 selects pixel sets for each angle. 

The data selecting unit 402 supplies the selected pixel 
10 sets to the error estimating unit 403. 

In step S405, the error estimating unit 403 calculates 
the correlation between the set of pixels centered on the 
pixel of interest, and the pixel sets selected for each 
angle. For example, the error estimating unit 4 03 
15 calculates the sum of absolute values of difference of the 

pixel values of the pixels of the set including the pixel of 
interest and the pixel values of the pixels at corresponding 
positions in other sets, for each angle. 

The angle of data continuity may be detected based on 
2 0 the correlation between pixel sets selected for each angle. 

The error estimating unit 403 supplies the information 
indicating the calculated correlation to the continuity 
direction derivation unit 404. 

In step S406, from position of the pixel set having the 
25 strongest correlation based on the correlation calculated in 
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the processing in step S405, the continuity direction 
derivation unit 404. detects the data continuity angle based 
on the reference axis in the input image which is image data 
that corresponds to the lost actual world 1 light signal 
continuity. For example, the continuity direction 
derivation unit 404 selects the smallest aggregate of the 
aggregate of absolute values of difference of pixel values, 
and detects the data continuity angle 0 from the position of 
the pixel set regarding which the selected aggregate has 
been calculated. 

The continuity direction derivation unit 404 outputs 
data continuity information indicating the angle of the data 
continuity that has been detected. 

In step S407, the data selecting unit 402 determines 
whether or not processing of all pixels has ended, and in 
the event that determination is made that processing of all 
pixels has not ended, the flow returns to step S401, a pixel 
of interest is selected from pixels not yet taken as the 
pixel of interest, and the above-described processing is 
repeated . 

In the event that determination is made in step S407 
that processing of all pixels has ended, the processing ends. 

Thus, the data continuity detecting unit 101 can detect 
the data continuity angle based on the reference axis in the 
image data, corresponding to the lost actual world 1 light 
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signal continuity. 

Note that an arrangement may be made wherein the data 
continuity detecting unit 101 of which the configuration is 
shown in Fig. 49 detects activity in the spatial direction 
5 of the input image with regard to the pixel of interest 

which is a pixel of interest in the frame of interest which 
is a frame of interest, extracts multiple pixel sets made up 
of a predetermined number of pixels in one row in the 
vertical direction or one row in the horizontal direction 

10 from the frame of interest and from each of frames before or 
after time-wise the frame of interest, for each angle and 
movement vector based on the pixel of interest and the 
space-directional reference axis, according to the detected 
activity, detects the correlation of the extracted pixel 

15 sets, and detects the data continuity angle in the time 

direction and spatial direction in the input image, based on 
this correlation. 

For example, as shown in Fig. 57, the data selecting 
unit 4 02 extracts multiple pixel sets made up of a 

20 predetermined number of pixels in one row in the vertical 

direction or one row in the horizontal direction from frame 
#n which is the frame of interest, frame #n-l, and frame 
#n+l, for each angle and movement vector based on the pixel 
of interest and the space-directional reference axis, 

25 according to the detected activity. 
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The frame #n-l is a frame which is previous to the 
frame #n time -wise, and the frame #n+l is a frame following 
the frame #n time-wise. That is to say, the frame #n-l, 
frame #n, and frame #n+l, are displayed in the order of 
5 frame #n-l, frame #n, and frame #n+l. 

The error estimating unit 403 detects the correlation 
of pixel sets for each single angle and single movement 
vector, with regard to the multiple sets of the pixels that 
have been extracted. The continuity direction derivation 
10 unit 4 04 detects the data continuity angle in the temporal 
direction and spatial direction in the input image which 
corresponds to the lost actual world 1 light signal 
continuity, based on the correlation of pixel sets, and 
outputs the data continuity information indicating the angle 
15 Next, description will be made regarding another 

embodiment example of the actual world estimating unit 102 
(Fig. 3) with reference to Fig. 58 through Fig. 88. 

Fig. 58 is a diagram for describing the principle of 
this embodiment example. 
20 As shown in Fig. 58, a signal (light intensity 

allocation) in the actual world 1, which is an image cast on 
the sensor 2, is represented with a predetermined function F 
Note that hereafter, with the description of this embodiment 
example, the signal serving as an image in the actual world 
25 1 is particularly referred to as a light signal, and the 
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function F is particularly referred to as a light signal 
function F. 

With this embodiment example, in the event that the 
light signal in the actual world 1 represented with the 
5 light signal function F has predetermined continuity, the 
actual world estimating unit 102 estimates the light signal 
function F by approximating the light signal function F with 
a predetermined function f using an input image (image data 
including continuity of data corresponding to continuity) 

10 from the sensor 2, and data continuity information (data 
continuity information corresponding to continuity of the 
input image data) from the data continuity detecting unit 
101. Note that with the description of this embodiment 
example, the function f is particularly referred to as an 

15 approximation function f, hereafter. 

In other words, with this embodiment example, the 
actual world estimating unit 102 approximates (describes) 
the image (light signal in the actual world 1) represented 
with the light signal function F using a model 161 (Fig. 4) 

20 represented with the approximation function f . Accordingly, 
hereafter, this embodiment example is referred to as a 
function approximating method. 

Now, description will be made regarding the background 
wherein the present applicant has invented the function 

2 5 approximating method, prior to entering the specific 
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description of the function approximating method. 

Fig. 59 is a diagram for describing integration effects 
in the case in which the sensor 2 is treated as a CCD. 

As shown in Fig. 59, multiple detecting elements 2-1 
5 are disposed on the plane of the sensor 2 . 

With the example in Fig. 59, a direction in parallel 
with a predetermined side of the detecting elements 2-1 is 
taken as the X direction, which is one direction in the 
spatial direction, and the a direction orthogonal to the X 
10 direction is taken as the Y direction, which is another 
direction in the spatial direction. Also, the direction 
perpendicular to the X-Y plane is taken as the direction t 
serving as the temporal direction. 

Also, with the example in Fig. 59, the spatial shape of 
15 each detecting element 2-1 of the sensor 2 is represented 

with a square of which one side is 1 in length. The shutter 
time (exposure time) of the sensor 2 is represented with 1. 

Further, with the example in Fig. 59, the center of one 
detecting element 2-1 of the sensor 2 is taken as the origin 
20 (position x = 0 in the X direction, and position y = 0 in 

the Y direction) in the spatial direction (X direction and Y 
direction) , and the intermediate point-in-time of the 
exposure time is taken as the origin (position t = 0 in the 
t direction) in the temporal direction (t direction) . 
25 In this case, the detecting element 2-1 of which the 
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center is in the origin (x = 0, y = 0) in the spatial 
direction subjects the light signal function F(x, y, t) to 
integration with a range between -0.5 and 0.5 in the X 
direction, range between -0.5 and 0.5 in the Y direction, 
5 and range between -0.5 and 0.5 in the t direction, and 
outputs the integral value thereof as a pixel value P. 

That is to say, the pixel value P output from the 
detecting element 2-1 of which the center is in the origin 
in the spatial direction is represented with the following 
10 Expression (38) . 



. . . (38) 

The other detecting elements 2-1 also output the pixel 
value P shown in Expression (3 8) by taking the center of the 
15 subject detecting element 2-1 as the origin in the spatial 
direction in the same way. 

Fig. 60 is a diagram for describing a specific example 
of the integration effects of the sensor 2. 



20 the X direction and Y direction of the sensor 2 (Fig. 59) . 

A portion 2301 of the light signal in the actual world 
1 (hereafter, such a portion is referred to as a region) 
represents an example of a region having predetermined 
continuity. 




In Fig. 60, the X direction and Y direction represent 
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Note that the region 2301 is a portion of the 
continuous light signal (continuous region) . On the other 
hand, in Fig. 60, the region 2301 is shown as divided into 
20 small regions (square regions) in reality. This is 
5 because of representing that the size of the region 2301 is 
equivalent to the size wherein the four detecting elements 
(pixels) of the sensor 2 in the X direction, and also the 
five detecting elements (pixels) of the sensor 2 in the Y 
direction are arrayed. That is to say, each of the 2 0 small 

10 regions (virtual regions) within the region 2301 is 
equivalent to one pixel. 

Also, a white portion shown in the drawing within the 
region 2301 represents a light signal corresponding to a 
fine line. Accordingly, the region 2301 has continuity in 

15 the direction wherein a fine line continues. Hereafter, the 
region 2301 is referred to as the fine-line-including actual 
world region 23 01. 

In this case, when the fine-line-including actual world 
region 2301 (a portion of a light signal in the actual world 

20 1) is detected by the sensor 2, region 2302 (hereafter, this 
is referred to as a fine- line- including data region 2302) of 
the input image (pixel values) is output from the sensor 2 
by integration effects. 

Note that each pixel of the fine-line-including data 

25 region 23 02 is represented as an image in the drawing, but 
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is data representing a predetermined value in reality. That 

is to say, the fine-line-including actual world region 2301 

is changed (distorted) to the fine-line-including data 

region 2302, which is divided into 20 pixels (20 pixels in 
5 total of 4 pixels in the X direction and also 5 pixels in 

the Y direction) each having a predetermined pixel value by 

the integration effects of the sensor 2. 

Fig. 61 is a diagram for describing another specific 

example (example different from Fig. 60) of the integration 
10 effects of the sensor 2 . 

In Fig. 61, the X direction and Y direction represent 

the X direction and Y direction of the sensor 2 (Fig. 59) . 
A portion (region) 2303 of the light signal in the 

actual world 1 represents another example (example different 
15 from the fine- line- including actual world region 2301 in Fig. 

60) of a region having predetermined continuity. 

Note that the region 2 3 03 is a region having the same 

size as the fine-line-including actual world region 2301. 

That is to say, the region 2303 is also a portion of the 
20 continuous light signal in the actual world 1 (continuous 

region) as with the fine-line-including actual world region 

2301 in reality, but is shown as divided into 20 small 

regions (square regions) equivalent to one pixel of the 

sensor 2 in Fig. 61. 
25 Also, the region 2303 includes a first portion having 
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predetermined first light intensity (value) , and a second 
portion edge having predetermined second light intensity 
(value) . Accordingly, the region 23 03 has continuity in the 
direction wherein the edges continue. Hereafter, the region 
5 2303 is referred to as the two-valued-edge-including actual 
world region 2 3 03. 

In this case, when the two-valued-edge-including actual 
world region 2303 (a portion of the light signal in the 
actual world 1) is detected by the sensor 2, a region 2304 

10 (hereafter, referred to as two-valued-edge- including data 

region 23 04) of the input image (pixel value) is output from 
the sensor 2 by integration effects. 

Note that each pixel value of the two-valued-edge- 
including data region 2 3 04 is represented as an image in the 

15 drawing as with the fine-line-including data region 2302, 
but is data representing a predetermined value in reality. 
That is to say, the two-valued-edge- including actual world 
region 2303 is changed (distorted) to the two-valued-edge- 
including data region 2304, which is divided into 20 pixels 

20 (20 pixels in total of 4 pixels in the X direction and also 
5 pixels in the Y direction) each having a predetermined 
pixel value by the integration effects of the sensor 2. 

Conventional image processing devices have regarded 
image data output from the sensor 2 such as the fine-line- 

25 including data region 2302, two- valued-edge- including data 
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region 2304, and the like as the origin (basis), and also 
have subjected the image data to the subsequent image 
processing. That is to say, regardless of that the image 
data output from the sensor 2 had been changed (distorted) 
5 to data different from the light signal in the actual world 
1 by integration effects, the conventional image processing 
devices have performed image processing on assumption that 
the data different from the light signal in the actual world 
1 is correct. 

10 As a result, the conventional image processing devices 

have provided a problem wherein based on the waveform (image 
data) of which the details in the actual world is distorted 
at the stage wherein the image data is output from the 
sensor 2, it is very difficult to restore the original 

15 details from the waveform. 

Accordingly, with the function approximating method, in 
order to solve this problem, as described above (as shown in 
Fig. 58) , the actual world estimating unit 102 estimates the 
light signal function F by approximating the light signal 

20 function F (light signal in the actual world 1) with the 
approximation function f based on the image data (input 
image) such as the fine-line-including data region 2302, and 
two-valued-edge- including data region 2304 output from the 
sensor 2 . 

25 Thus, at a later stage than the actual world estimating 
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unit 102 (in this case, the image generating unit 103 in Fig. 
3) , the processing can be performed by taking the image data 
wherein integration effects are taken into consideration, 
i.e., image data that can be represented with the 
5 approximation function f as the origin. 

Hereafter, description will be made independently 
% . regarding three specific methods (first through third 
function approximating methods) , of such a function 
approximating method with reference to the drawings. 
10 First, description will be made regarding the first 

function approximating method with reference to Fig. 62 
through Fig. 76. 

Fig. 62 is a diagram representing the fine-line- 
including actual world region 2301 shown in Fig. 60 
15 described above again. 

In Fig. 62, the X direction and Y direction represent 
the X direction and Y direction of the sensor 2 (Fig. 59) . 

The first function approximating method is a method for 
approximating a one -dimensional waveform (hereafter, such a 
2 0 waveform is referred to as an X cross -sectional waveform 
F(x)) wherein the light signal function F(x, y, t) 
corresponding to the fine-line- including actual world region 
2301 such as shown in Fig. 62 is projected in the X 
direction (direction of an arrow 2311 in the drawing) , with 
25 the approximation function f (x) serving as an n-dimensional 
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(n is an arbitrary integer) polynomial, for example. 
Accordingly, hereafter, the first function approximating 
method is particularly referred to as a one-dimensional 
approximat ing method . 
5 Note that with the one -dimensional approximating method, 

the X cross-sectional waveform F (x) , which is to be 
approximated, is not restricted to a waveform corresponding 
to the fine-line-including actual world region 2301 in Fig. 
62, of course. That is to say, as described later, with the 

10 one -dimensional approximating method, any waveform can be 

approximated as long as the X cross-sectional waveform F (x) 
corresponds to the light signals in the actual world 1 
having continuity. 

Also, the direction of the projection of the light 

15 signal function F(x, y, t) is not restricted to the X 

direction, or rather the Y direction or t direction may be 
employed. That is to say, with the one -dimensional 
approximating method, a function F (y) wherein the light 
signal function F(x, y, t) is projected in the Y direction 

2 0 may be approximated with a predetermined approximation 

function f (y) , or a function F(t) wherein the light signal 
function F (x, y, t) is projected in the t direction may be 
approximated with a predetermined approximation function f 
(t) . 

25 More specifically, the one-dimensional approximating 
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method is a method for approximating, for example, the X 
cross-sectional waveform F (x) with the approximation 
function f (x) serving as an n-dimensional polynomial such as 
shown in the following Expression (39) . 

n 

f (X) = W 0 +W 1 X+W 2 X+- ' -+W n X n = Z WjX' 

i=0 

. . . (39) 

That is to say, with the one-dimensional approximating 
method, the actual world estimating unit 102 estimates the X 
cross-sectional waveform F(x) by calculating the coefficient 
(features) Wi of x 1 in Expression (39) . 

This calculation method of the features w± is not 
restricted to a particular method, for example, the 
following first through third methods may be employed. 

That is to say, the first method is a method that has 
been employed so far. 

On the other hand, the second method is a method that 
has been newly invented by the present applicant, which is a 
method that considers continuity in the spatial direction as 
to the first method. 

However, as described later, with the first and second 
methods, the integration effects of the sensor 2 are not 
taken into consideration. Accordingly, an approximation 
function f (x) obtained by substituting the features wi 
calculated by the first method or the second method for the 
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above Expression (39) is an approximation function regarding 
an input image, but strictly speaking, cannot be referred to 
as the approximation function of the X cross-sectional 
waveform F (x) . 

5 Consequently, the present applicant has invented the 

third method that calculates the features w ± further in 
light of the integration effects of the sensor 2 as to the 
second method. An approximation function f (x) obtained by 
substituting the features w± calculated with this third 

10 method for the above Expression (39) can be referred to as 

the approximation function of the X cross-sectional waveform 
F(x) in that the integration effects of the sensor 2 are 
taken into consideration. 

Thus, strictly speaking, the first method and the 

15 second method cannot be referred to as the one-dimensional 
approximating method, and the third method alone can be 
referred to as the one-dimensional approximating method. 

In other words, as shown in Fig. 63, the second method 
is different from the one -dimensional approximating method. 

20 That is to say, Fig. 63 is a diagram for describing the 
principle of the embodiment corresponding to the second 
method. 

As shown in Fig. 63, with the embodiment corresponding 
to the second method, in the event that the light signal in 
25 the actual world 1 represented with the light signal 
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function F has predetermined continuity, the actual world 
estimating unit 102 does not approximate the X cross- 
sectional waveform F (x) with an input image (image data 
including continuity of data corresponding to continuity) 
5 from the sensor 2, and data continuity information (data 

continuity information corresponding to continuity of input 
w image data) from the data continuity detecting unit 101, but 

approximates the input image from the sensor 2 with a 
predetermined approximation function f 2 (x) . 

10 Thus, it is hard to say that the second method is a 

method having the same level as the third method in that 
approximation of the input image alone is performed without 
considering the integral effects of the sensor 2. However, 
the second method is a method superior to the conventional 

15 first method in that the second method takes continuity in 
the spatial direction into consideration. 

Hereafter, description will be made independently 
regarding the details of the first method, second method, 
and third method in this order. 

2 0 Note that hereafter, in the event that the respective 

approximation functions f (x) generated by the first method, 
second method, and third method are distinguished from that 
of the other method, they are particularly referred to as 
approximation function fi (x) , approximation function f 2 (x) , 

25 and approximation function f 3 (x) respectively. 
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First, description will be made regarding the details 
of the first method. 

With the first method, on condition that the 
approximation function f x (x) shown in the above Expression 

(39) holds within the fine-line-including actual world 
region 2301 in Fig. 64, the following prediction equation 

(40) is defined. 

P(x.y) =f 1 (x)+e 

. . . (40) 

In Expression (40) , x represents a pixel position 
relative as to the X direction from a pixel of interest. y 
represents a pixel position relative as to the Y direction 
from the pixel of interest. e represents a margin of error. 
Specifically, for example, as shown in Fig. 64, let us say 
that the pixel of interest is the second pixel in the X 
direction from the left, and also the third pixel in the Y 
direction from the bottom in the drawing, of the fine-line- 
including data region 2302 (data of which the fine-line- 
including > actual world region 23 01 (Fig. 62) is detected by 
the sensor 2, and output) . Also, let us say that the center 
of the pixel of interest is the origin (0, 0) , and a 
coordinates system (hereafter, referred to as a pixel-of- 
interest coordinates system) of which axes are an x axis and 
y axis respectively in parallel with the X direction and Y 
direction of the sensor 2 (Fig. 59) is set. In this case, 
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the coordinates value (x, y) of the pixel -of - interest 
coordinates system represents a relative pixel position. 

Also, in Expression (40) , P (x, y) represents a pixel 
value in the relative pixel positions (x, y) . Specifically, 
in this case, the P (x, y) within the fine-line-including 
data region 2302 is such as shown in Fig. 65. 

Fig. 65 represents this pixel value P (x, y) in a 
graphic manner. 

In Fig. 65, the respective vertical axes of the graphs 
represent pixel values, and the horizontal axes represent a 
relative position x in the X direction from the pixel of 
interest. Also, in the drawing, the dashed line in the 
first graph from the top represents an input pixel value P 
(x, -2) , the broken triple-dashed line in the second graph 
from the top represents an input pixel value P (x, -1) , the 
solid line in the third graph from the top represents an 
input pixel value P (x, 0) , the broken line in the fourth 
graph from the top represents an input pixel value P (x, 1) , 
and the broken double-dashed line in the fifth graph from 
the top (the first from the bottom) represents an input 
pixel value P (x, 2) respectively. 

Upon the 20 input pixel values P (x, -2) , P (x, -1) , P 
(x, 0), P (x, 1), and P (x, 2) (however, x is any one 
integer value of -1 through 2) shown in Fig. 65 being 
substituted for the above Expression (40) respectively, 20 
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equations as shown in the following Expression (41) are 
generated. Note that each e k (k is any one of integer 
values 1 through 20) represents a margin of error. 
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. . . (41) 

Expression (41) is made up of 2 0 equations, so in the 
event that the number of the features w± of the 
approximation function f x (x) is less than 20, i.e., in the 
event that the approximation function fi (x) is a polynomial 
having the number of dimensions less than 19, the features 



wi can be calculated using the least square method, for 
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example. Note that the specific solution of the least 
square method will be described later. 

For example, if we say that the number of dimensions of 
the approximation function f x (x) is five, the approximation 
function fi (x) calculated with the least square method 
using Expression (41) (the approximation function fi (x) 
generated by the calculated features w ± ) becomes a curve 
shown in Fig. 66. 

Note that in Fig. 66, the vertical axis represents 
pixel values, and the horizontal axis represents a relative 
position x from the pixel of interest. 

That is to say, for example, if we supplement the 
respective 20 pixel values P (x, y) (the respective input 
pixel values P (x, -2), P (x, -1), P (x, 0), P (x, 1), and P 
(x, 2) shown in Fig. 65) making up the fine-line-including 
data region 2302 in Fig. 64 along the x axis without any 
modification (if we regard a relative position y in the Y 
direction as constant, and overlay the five graphs shown in 
Fig. 65) , multiple lines (dashed line, broken triple-dashed 
line, solid line, broken line, and broken double-dashed 
line) in parallel with the x axis, such as shown in Fig. 66, 
are distributed. 

However, in Fig. 66, the dashed line represents the 
input pixel value P (x, -2) , the broken triple-dashed line 
represents the input pixel value P (x, -1) , the solid line 
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represents the input pixel value P (x, 0) , the broken line 
represents the input pixel value P (x, 1) , and the broken 
double-dashed line represents the input pixel value P (x, 2) 
respectively. Also, in the event of the same pixel value, 
5 lines more than 2 lines are overlaid in reality, but in Fig. 
66, the lines are drawn so as to distinguish each line, and 
so as not to overlay each line. 
* The respective 20 input pixel values (P (x, -2) , P (x, 

-1), P (x, 0), P (x, 1), and P (x, 2)) thus distributed, and 

10 a regression curve (the approximation function fi (x) 

obtained by substituting the features w± calculated with the 
least square method for the above Expression (38) ) so as to 
minimize the error of the value fi (x) become a curve 
(approximation function f ± (x) ) shown in Fig. 66. 

15 Thus, the approximation function f ± (x) represents 

nothing but a curve connecting in the X direction the means 
of the pixel values (pixel values having the same relative 
position x in the X direction from the pixel of interest) P 
(x, -2), P (x, -1), P (x, 0), P (x, 1), and P (x, 2) in the 

2 0 Y direction. That is to say, the approximation function fi 
(x) is generated without considering continuity in the 
spatial direction included in the light signal. 

For example, in this case, the fine-line-including 
actual world region 23 01 (Fig. 62) is regarded as a subject 

25 to be approximated. This fine-line-including actual world 
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region 2301 has continuity in the spatial direction, which 
is represented with a gradient G F , such as shown in Fig. 67. 
Note that in Fig. 67, the X direction and Y direction 
represent the X direction and Y direction of the sensor 2 
(Fig. 59) . 

Accordingly, the data continuity detecting unit 101 
(Fig. 58) can output an angle 0 (angle 9 generated between 
the direction of data continuity represented with a gradient 
G f corresponding to the gradient G F/ and the X direction) 
such as shown in Fig. 67 as data continuity information 
corresponding to the gradient G F as continuity in the 
spatial direction 

However, with the first method, the data continuity 
information output from the data continuity detecting unit 
101 is not employed at all. 

In other words, such as shown in Fig. 67, the direction 
of continuity in the spatial direction of the fine-line- 
including actual world region 2301 is a general angle 9 
direction. However, the first method is a method for 
calculating the features w ± of the approximation function fi 
(x) on assumption that the direction of continuity in the 
spatial direction of the fine-line-including actual world 
region 2301 is the Y direction (i.e., on assumption that the 

angle 0 is 90°) . 

Consequently, the approximation function f x (x) becomes 
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a function of which the waveform gets dull, and the detail 
decreases than the original pixel value. In other words, 
though not shown in the drawing, with the approximation 
function f x (x) generated with the first method, the 
5 waveform thereof becomes a waveform different from the 
actual X cross-sectional waveform F (x) . 

To this end, the present applicant has invented the 
second method for calculating the features w± by further 
taking continuity in the spatial direction into 
10 consideration (utilizing the angle 6) as to the first method. 

That is to say, the second method is a method for 
calculating the features w ± of the approximation function f 2 
(x) on assumption that the direction of continuity of the 
fine-line-including actual world region 2301 is a general 

15 angle 0 direction. 

Specifically, for example, the gradient G f representing 
continuity of data corresponding to continuity in the 
spatial direction is represented with the following 
Expression (42) . 

G f = tan G = 

. . . (42) 

Note that in Expression (42) , dx represents the amount 
of fine movement in the X direction such as shown in Fig. 67, 
dy represents the amount of fine movement in the Y direction 



20 
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as to the dx such as shown in Fig. 67. 

In this case, if we define the shift amount C x (y) as 
shown in the following Expression (43) , with the second 
method, an equation corresponding to Expression (40) 
5 employed in the first method becomes such as the following 
Expression (44) . 




. . - (43) 

P(x, y) =f 2 (x-C x (y))+e 

10 • ^ (44) 

That is to say, Expression (40) employed in the first 
method represents that the position x in the X direction of 
the pixel center position (x, y) is the same value regarding 
the pixel value P (x, y) of any pixel positioned in the same 

15 position. In other words, Expression (40) represents that 
pixels having the same pixel value continue in the Y 
direction (exhibits continuity in the Y direction) . 

On the other hand, Expression (44) employed in the 
second method represents that the pixel value P (x, y) of a 

20 pixel of which the center position is (x, y) is not 

identical to the pixel value (approximate equivalent to f 2 
(x) ) of a pixel positioned in a place distant from the pixel 
of interest (a pixel of which the center position is the 
origin (0, 0)) in the X direction by x, and is the same 
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value as the pixel value (approximate equivalent to f 2 (x + 
C x (y) ) of a pixel positioned in a place further distant 
from the pixel thereof in the X direction by the shift 
amount C x (y) (pixel positioned in a place distant from the 
5 pixel of interest in the X direction by x + C x (y) ) . In 
other words, Expression (44) represents that pixels having 
the same pixel value continue in the angle 0 direction 
corresponding to the shift amount C x (y) (exhibits 
continuity in the general angle 0 direction) . 

10 Thus, the shift amount C x (y) is the amount of 

correction considering continuity (in this case, continuity 
represented with the gradient G F in Fig. 67 (strictly 
speaking, continuity of data represented with the gradient 
G f ) ) in the spatial direction, and Expression (44) is 

15 obtained by correcting Expression (4 0) with the shift amount 
C x (y) . 

In this case, upon the 20 pixel values P (x, y) 
(however, x is any one integer value of -1 through 2, and y 
is any one integer value of -2 through 2) of the fine-line- 
2 0 including data region 23 02 shown in Fig. 64 being 

substituted for the above Expression (44) respectively, 20 
equations as shown in the following Expression (45) are 
generated . 
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. . . (45) 

Expression (4 5) is made up of 2 0 equations, as with the 
above Expression (41) . Accordingly, with the second method, 
5 as with the first method, in the event that the number of 

the features wi of the approximation function f 2 (x) is less 
than 20, i.e., the approximation function f 2 (x) is a 
polynomial having the number of dimensions less than 19, the 
features wi can be calculated with the least square method, 
10 for example. Note that the specific solution regarding the 



least square method will be described later. 



- 193 - 

S04P0850 

For example, if we say that the number of dimensions of 
the approximation function f 2 (x) is five as with the first 
method, with the second method, the features w ± are 
calculated as follows. 
5 That is to say, Fig. 68 represents the pixel value P (x, 

y) shown in the left side of Expression (45) in a graphic 
manner. The respective five graphs shown in Fig. 68 are 
basically the same as shown in Fig. 65. 

As shown in Fig. 68, the maximal pixel values (pixel 
10 values corresponding to fine lines) are continuous in the 
direction of continuity of data represented with the 
gradient G f . 

Consequently, with the second method, if we supplement 
the respective input pixel values P (x, -2) , P (x, -1) , P (x, 

15 0) , P (x, 1), and P (x, 2) shown in Fig. 68, for example, 
along the x axis, we supplement the pixel values after the 
pixel values are changed in the states shown in Fig. 69 
instead of supplementing the pixel values without any 
modification as with the first method (let us assume that y 

2 0 is constant, and the five graphs are overlaid in the states 
shown in Fig. 68) . 

That is to say, Fig. 6 9 represents a state wherein the 
respective input pixel values P (x, -2) , P (x, -1) , P (x, 0) , 
P (x, 1) , and P (x, 2) shown in Fig. 68 are shifted by the 

25 shift amount C x (y) shown in the above Expression (43) . In 
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other words, Fig. 6 9 represents a state wherein the five 
graphs shown in Fig. 68 are moved as if the gradient G F 
representing the actual direction of continuity of data were 
regarded as a gradient G F 1 (in the drawing, a straight line 
5 made up of a dashed line were regarded as a straight line 
made up of a solid line) . 

In the states in Fig. 69, if we supplement the 
respective input pixel values P (x, -2) , P (x, -1) , P (x, 0) , 
P (x, 1) , and P (x, 2) , for example, along the x axis (in 
10 the states shown in Fig. 69, if we overlay the five graphs), 
multiple lines (dashed line, broken triple-dashed line, 
solid line, broken line, and broken double-dashed line) in 
parallel with the x axis, such as shown in Fig. 70, are 
distributed. 

15 Note that in Fig. 70, the vertical axis represents 

pixel values, and the horizontal axis represents a relative 
position x from the pixel of interest. Also, the dashed 
line represents the input pixel value P (x, -2), the broken 
triple-dashed line represents the input pixel value P (x, - 

20 1) , the solid line represents the input pixel value P (x, 0) , 
the broken line represents the input pixel value P (x, 1) , 
and the broken double-dashed line represents the input pixel 
value P (x, 2) respectively. Further, in the event of the 
same pixel value, lines more than 2 lines are overlaid in 

25 reality, but in Fig. 70, the lines are drawn so as to 
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distinguish each line, and so as not to overlay each line. 

The respective 2 0 input pixel values P (x, y) (however, 
x is any one integer value of -1 through 2, and y is any one 
integer value of -2 through 2) thus distributed, and a 
5 regression curve (the approximation function f 2 (x) obtained 
by substituting the features w± calculated with the least 
square method for the above Expression (38) ) to minimize the 
error of the value f 2 (x + C x (y) ) become a curve f 2 (x) 
shown in the solid line in Fig. 70. 

10 Thus, the approximation function f 2 (x) generated with 

the second method represents a curve connecting in the X 
direction the means of the input pixel values P (x, y) in 
the angle 0 direction (i.e., direction of continuity in the 
general spatial direction) output from the data continuity 

15 detecting unit 101 (Fig. 58) . 

On the other hand, as described above, the 
approximation function fi (x) generated with the first 
method represents nothing but a curve connecting in the X 
direction the means of the input pixel values P (x, y) in 

20 the Y direction (i.e., the direction different from the 
continuity in the spatial direction) . 

Accordingly, as shown in Fig. 70, the approximation 
function f 2 (x) generated with the second method becomes a 
function wherein the degree of dullness of the waveform 

25 thereof decreases, and also the degree of decrease of the 
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detail as to the original pixel value decreases less than 
the approximation function fi (x) generated with the first 
method. In other words, though not shown in the drawing, 
with the approximation function f 2 (x) generated with the 
5 second method, the waveform thereof becomes a waveform 

closer to the actual X cross-sectional waveform F (x) than 
the approximation function fi (x) generated with the first 
method . 

However, as described above, the approximation function 

10 f 2 (x) is a function considering continuity in the spatial 
direction, but is nothing but a function generated wherein 
the input image (input pixel value) is regarded as the 
origin (basis) . That is to say, as shown in Fig. 63 
described above, the approximation function f 2 (x) is 

15 nothing but a function that approximated the input image 

different from the X cross-sectional waveform F (x) , and it 
is hard to say that the approximation function f 2 (x) is a 
function that approximated the X cross- sect ional waveform 
F (x) . In other words, the second method is a method for 

2 0 calculating the features w± on assumption that the above 
Expression (44) holds, but does not take the relation in 
Expression (38) described above into consideration (does not 
consider the integration effects of the sensor 2) . 

Consequently, the present applicant has invented the 

25 third method that calculates the features w± of the 
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approximation function f 3 (x) by further taking the 
integration effects of the sensor 2 into consideration as to 
the second method. 

That is to say, the third method is a method that 
5 introduces the concept of a spatial mixture or time mixture. 
Now, considering both spatial mixture and time mixture will 
complicate the description, so only spatial mixture will be 
considered here of spatial mixture and time mixture, and 
time mixture will be ignored. 
10 Description will be made regarding spatial mixture with 

reference to Fig. 71 prior to description of the third 
method . 

In Fig. 71, a portion 2321 (hereafter, referred to as a 
region 2321) of a light signal in the actual world 1 
15 represents a region having the same area as one detecting 
element (pixel) of the sensor 2. 

Upon the sensor 2 detecting the region 2321, the sensor 
2 outputs a value (one pixel value) 2322 obtained by the 
region 2321 being subjected to integration in the temporal 
20 and spatial directions (X direction, Y direction, and t 

direction) . Note that the pixel value 2322 is represented 
as an image in the drawing, but is actually data 
representing a predetermined value. 

The region 2321 in the actual world 1 is clearly 
25 classified into a light signal (white region in the drawing) 
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corresponding to the foreground (the above fine line, for 
example) , and a light signal (black region in the drawing) 
corresponding to the background. 

On the other hand, the pixel value 2322 is a value 
5 obtained by the light signal in the actual world 1 

corresponding to the foreground and the light signal in the 
actual world 1 corresponding to the background being 
subjected to integration. In other words, the pixel value 
2322 is a value corresponding to a level wherein the light 

10 level corresponding to the foreground and the light level 
corresponding to the background are spatially mixed. 

Thus, in the event that a portion corresponding to one 
pixel (detecting element of the sensor 2) of the light 
signals in the actual world 1 is not a portion where the 

15 light signals having the same level are spatially uniformly 
distributed, but a portion where the light signals having a 
different level such as a foreground and background are 
distributed, upon the region thereof being detected by the 
sensor 2, the region becomes one pixel value as if the 

2 0 different light levels were spatially mixed by the 

integration effects of the sensor 2 (integrated in the 
spatial direction) . Thus, a region made up of pixels in 
which an image (light signals in the actual world 1) 
corresponding to a foreground, and an image (light signals 

2 5 in the actual world 1) corresponding to a background are 
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subjected to spatial integration, that is, mixed as it were, 
which is spatial mixing, is referred to as a spatial mixed 
region here. 

Accordingly, with the third method, the actual world 
5 estimating unit 102 (Fig. 58) estimates the X cross- 
sectional waveform F (x) representing the original region 
2321 in the actual world 1 (of the light signals in the 
actual world 1, the portion 2321 corresponding to one pixel 
of the sensor 2) by approximating the X cross- sect ional 

10 waveform F (x) with the approximation function f 3 (x) serving 
as a one-dimensional polynomial such as shown in Fig. 72. 

That is to say, Fig. 72 represents an example of the 
approximation function f 3 (x) corresponding to the pixel 
value 2322 serving as a spatial mixed region (Fig. 71), i.e., 

15 the approximation function f 3 (x) that approximates the X 
cross-sectional waveform F (x) corresponding to the solid 
line within the region 2331 in the actual world 1 (Fig. 71) . 
In Fig. 72, the axis in the horizontal direction in the 
drawing represents an axis in parallel with the side from 

2 0 the upper left end x s to lower right end x e of the pixel 

corresponding to the pixel value 2322 (Fig. 71), which is 
taken as the x axis. The axis in the vertical direction in 
the drawing is taken as an axis representing pixel values. 

In Fig. 72, the following Expression (46) is defined on 

2 5 condition that the result obtained by subjecting the 



P = J" x * e f 3 (x)dx+e 
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approximation function f 3 (x) to integration in a range 
(pixel width) from the x s to the x e is generally identical 
with the pixel values P (x, y) output from the sensor 2 
(dependent on a margin of error e alone) . 
> x p 

e (w 0 +w 1 x+w 2 x 2 +- • • +w n x n )dx+e 
= w 0 (x e -x s ) + • • ■ +w nH +w n Xe n ~* s +e 

. . . (46) 

In this case, the features Wi of the approximation 
function f 3 (x) are calculated from the 20 pixel values P (x, 
y) (however, x is any one integer value of -1 through 2, and 
10 y is any one integer value of -2 through 2) of the fine- 
line-including data region 2302 shown in Fig. 67, so the 
pixel value P in Expression (46) becomes the pixel values P 
(x, y) . 

Also, as with the second method, it is necessary to 
15 take continuity in the spatial direction into consideration, 
and accordingly, each of the start position x s and end 
position x e in the integral range in Expression (46) is 
dependent upon the shift amount C x (y) . That is to say, 
each of the start position x s and end position x e of the 
20 integral range in Expression (46) is represented such as the 
following Expression (47) . 
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x s = x-C x (y) -0.5 
x e = x-C x (y) +0.5 

. . . (47) 

In this case, upon each pixel value of the fine-line- 
including data region 2302 shown in Fig. 67, i.e., each of 
5 the input pixel values P (x, -2) , P (x, -1) , P (x, 0) , P (x, 
1) , and P (x, 2) (however, x is any one integer value of -1 
through 2) shown in Fig. 68 being substituted for the above 
Expression (46) (the integral range is the above Expression 
(47)), 20 equations shown in the following Expression (48) 
10 are generated. 



3 
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Expression (48) is made up of 20 equations as with the 
above Expression (45) . Accordingly, with the third method 
as with the second method, in the event that the number of 
the features wi of the approximation function f 3 (x) is less 
than 20, i.e., in the event that the approximation function 
f 3 (x) is a polynomial having the number of dimensions less 
than 19, for example, the features w± may be calculated with 
the least square method. Note that the specific solution of 
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the least square method will be described later. 

For example, if we say that the number of dimensions of 
the approximation function f 3 (x) is five, the approximation 
function f 3 (x) calculated with the least square method 
5 using Expression (48) (the approximation function f 3 (x) 

generated with the calculated features w ± ) becomes a curve 
shown with the solid line in Fig. 73. 

Note that in Fig. 73, the vertical axis represents 
pixel values, and the horizontal axis represents a relative 

10 position x from the pixel of interest. 

As shown in Fig. 73, in the event that the 
approximation function f 3 (x) (a curve shown with a solid 
line in the drawing) generated with the third method is 
compared with the approximation function f 2 (x) (a curve 

15 shown with a dashed line in the drawing) generated with the 
second method, a pixel value at x = 0 becomes great, and 
also the gradient of the curve creates a steep waveform. 
This is because details increase more than the input pixels, 
resulting in being unrelated to the resolution of the input 

20 pixels. That is to say, we can say that the approximation 
function f 3 (x) approximates the X cross- sectional waveform 
F (x) . Accordingly, though not shown in the drawing, the 
approximation function f 3 (x) becomes a waveform closer to 
the X cross-sectional waveform F (x) than the approximation 

25 function f 2 (x) . 
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Fig. 74 represents a configuration example of the 
actual world estimating unit 102 employing such a one- 
dimensional approximating method. 

In Fig. 74, the actual world estimating unit 102 
5 estimates the X cross-sectional waveform F (x) by calculating 
the features w± using the above third method (least square 
method) for example, and generating the approximation 
function f (x) of the above Expression (39) using the 
calculated features wi . 
10 As shown in Fig. 74, the actual world estimating unit 

102 includes a conditions setting unit 2331, input image 
storage unit 2332, input pixel value acquiring unit 2333, 
integral component calculation unit 2334, normal equation 
generating unit 2335, and approximation function generating 
15 unit 2336. 

The conditions setting unit 2331 sets a pixel range 
(hereafter, referred to as a tap range) used for estimating 
the X cross-sectional waveform F (x) corresponding to a pixel 
of interest, and the number of dimensions n of the 
20 approximation function f (x) . 

The input image storage unit 2332 temporarily stores an 
input image (pixel values) from the sensor 2. 

The input pixel value acquiring unit 2333 acquires, of 
the input images stored in the input image storage unit 2332, 
2 5 an input image region corresponding to the tap range set by 
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the conditions setting unit 2231, and supplies this to the 
normal equation generating unit 2335 as an input pixel value 
table. That is to say, the input pixel value table is a 
table in which the respective pixel values of pixels 
5 included in the input image region are described. Note that 
a specific example of the input pixel value table will be 
described later. 

Now, the actual world estimating unit 102 calculates 
the features wi of the approximation function f (x) with the 
10 least square method using the above Expression (46) and 

Expression (47) here, but the above Expression (4 6) can be 
represented such as the following Expression (49) . 

P(x,y) 

" (x-C x (y) +0. 5) i+1 - (x-C x (y) -0. 5) ,+1 

= | 0 WiX — m +e 

n 

= Z Wj xSj (x s , x e ) +e 

i=0 

. . . (49) 

15 In Expression (49) , Si (x S/ x e ) represents the integral 

components of the i-dimensional term. That is to say, the 
integral components Si (x s , x e ) are shown in the following 
Expression (50) . 

i+1_ i+1 

Sj Cx s , x e ) = r^pj 

20 ... (50) 
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The integral component calculation unit 2334 calculates 
the integral components S± (x s , x e ) . 

Specifically, the integral components Si (x s , x e ) 
(however, the value x s and value x e are values shown in the 
5 above Expression (46) ) shown in Expression (50) may be 

calculated as long as the relative pixel positions (x, y) , 
shift amount C x (y) , and i of the i -dimensional terms are 
known. Also, of these, the relative pixel positions (x, y) 
are determined by the pixel of interest and the tap range, 

10 the shift amount C x (y) is determined by the angle 6 (by the 
above Expression (41) and Expression (43) ) , and the range of 
i is determined by the number of dimensions n, respectively. 

Accordingly, the integral component calculation unit 
2334 calculates the integral components S± (x s , x e ) based on 

15 the tap range and the number of dimensions set by the 

conditions setting unit 2331, and the angle 0 of the data 
continuity information output from the data continuity 
detecting unit 101, and supplies the calculated results to 
the normal equation generating unit 2335 as an integral 

2 0 component table. 

The normal equation generating unit 233 5 generates the 
above Expression (46), i.e., a normal equation in the case 
of obtaining the features wi of the right side of Expression 
(4 9) with the least square method using the input pixel 

2 5 value table supplied from the input pixel value acquiring 
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unit 2333, and the integral component table supplied from 
the integral component calculation unit 2334, and supplies 
this to the approximation function generating unit 2 336 as a 
normal equation table. Note that a specific example of a 
5 normal equation will be described later. 

The approximation function generating unit 233 6 
calculates the respective features wi of the above 
Expression (49) (i.e., the respective coefficients w± of the 
approximation function f (x) serving as a one -dimensional 

10 polynomial) by solving a normal equation included in the 
normal equation table supplied from the normal equation 
generating unit 2335 using the matrix solution, and outputs 
these to the image generating unit 103 . 

Next, description will be made regarding the actual 

15 world estimating processing (processing in step S102 in Fig. 
29) of the actual world estimating unit 102 (Fig. 74) which 
employs the one-dimensional approximating method with 
reference to the flowchart in Fig. 75. 

For example, let us say that an input image, which is a 

20 one-frame input image output from the sensor 2, including 
the fine-line-including data region 2302 in Fig. 60 
described above has been already stored in the input image 
storage unit 2332. Also, let us say that the data 
continuity detecting unit 101 has subjected, at the 

25 continuity detection processing in step S101 (Fig. 29) , the 
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fine-line-including data region 2302 to the processing 
thereof, and has already output the angle 0 as data 
continuity information . 

In this case, the conditions setting unit 2331 sets 
5 conditions (a tap range and the number of dimensions) in 
step S2301 in Fig. 75. 

For example, let us say that a tap range 23 51 shown in 
Fig. 76 is set, and 5 dimensions are set as the number of 
dimensions . 

10 That is to say, Fig. 76 is a diagram for describing an 

example of a tap range. In Fig. 76, the X direction and Y 
direction are the X direction and Y direction of the sensor 
2 (Fig. 59) respectively. Also, the tap range 2351 
represents a pixel group made up of 20 pixels in total (20 

15 squares in the drawing) of 4 pixels in the X direction, and 
also 5 pixels in the Y direction. 

Further, as shown in Fig. 76, let us say that a pixel 
of interest is set at the second pixel from the left and 
also the third pixel from the bottom in the drawing, of the 

20 tap range 2351. Also, let us say that each pixel is denoted 
with a number 1 such as shown in Fig. 76 (1 is any integer 
value of 0 through 19) according to the relative pixel 
positions (x, y) from the pixel of interest (a coordinate 
value of a pixel-of -interest coordinates system wherein the 

25 center (0, 0) of the pixel of interest is taken as the 
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origin) . 

Now, description will return to Fig. 75, wherein in 
step S2302, the conditions setting unit 2331 sets a pixel of 
interest . 

In step S2303, the input pixel value acquiring unit 
2333 acquires an input pixel value based on the condition 
(tap range) set by the conditions setting unit 2331, and 
generates an input pixel value table. That is to say, in 
this case, the input pixel value acquiring unit 2333 
acquires the fine-line-including data region 2302 (Fig. 64), 
and generates a table made up of 2 0 input pixel values P (1) 
as an input pixel value table. 

Note that in this case, the relation between the input 
pixel values P (1) and the above input pixel values P (x, y) 
is a relation shown in the following Expression (51) . 
However, in Expression (51) , the left side represents the 
input pixel values P (1) , and the right side represents the 
input pixel values P (x, y) . 
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In step S2304, the integral component calculation unit 
2334 calculates integral components based on the conditions 
5 (a tap range and the number of dimensions) set by the 
conditions setting unit 2331, and the data continuity 
information (angle 0) supplied from the data continuity 
detecting unit 101, and generates an integral component 
table . 

10 In this case, as described above, the input pixel 

values are not P (x, y) but P (1) , and are acquired as the 
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value of a pixel number 1, so the integral component 
calculation unit 2334 calculates the above integral 
components Si (x S/ x e ) in Expression (50) as a function of 1 
such as the integral components Si (1) shown in the left 
side of the following Expression (52) . 

Sj(l) =Sj(x s ,x e ) 

. . . (52) 

Specifically, in this case, the integral components Si 
(1) shown in the following Expression (53) are calculated. 
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Note that in Expression (53) , the left side represents 
the integral components Si (1) , and the right side 
represents the integral components Si (x s/ x e ) . That is to 
5 say, in this case, i is 0 through 5, and accordingly, the 
120 Si (1) in total of the 20 S 0 (1), 20 Si (1), 20 S 2 (1), 
20 S 3 (1) , 20 S 4 (1) , and 20 S 5 (1) are calculated. 

More specifically, first the integral component 
calculation unit 2334 calculates each of the shift amounts 

10 C x (-2), C x (-1), C x (1), and C x (2) using the angle 0 

supplied from the data continuity detecting unit 101. Next, 
the integral component calculation unit 2334 calculates each 
of the 2 0 integral components Si (x s , x e ) shown in the right 
side of Expression (52) regarding each of i = 0 through 5 

15 using the calculated shift amounts C x (-2), C x (-1), C x (1), 
and C x (2) . That is to say, the 120 integral components Si 
(x s , x e ) are calculated. Note that with this calculation of 
the integral components Si (x s , x e ) , the above Expression 
(50) is used. Subsequently, the integral component 

20 calculation unit 2334 converts each of the calculated 120 
integral components S± (x s , x e ) into the corresponding 
integral components S± (1) in accordance with Expression 
(53) , and generates an integral component table including 
the converted 12 0 integral components Si (1) . 

25 Note that the sequence of the processing in step S2303 



10 



15 
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and the processing in step S2304 is not restricted to the 
example in Fig. 75, the processing in step S2304 may be 
executed first, or the processing in step S23 03 and the 
processing in step S2304 may be executed simultaneously. 

Next, in step S2305, the normal equation generating 
unit 2335 generates a normal equation table based on the 
input pixel value table generated by the input pixel value 
acquiring unit 2333 at the processing in step S2303, and the 
integral component table generated by the integral component 
calculation unit 2334 at the processing in step S2304. 

Specifically, in this case, the features w ± of the 
following Expression (54) corresponding to the above 
Expression (4 9) are calculated using the least square method. 
A normal equation corresponding to this is represented as 
the following Expression (55) . 



PCI) = I WjXSjCD+e 

i=0 
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Note that in Expression (55) , L represents the maximum 
value of the pixel number 1 in the tap range. n represents 
the number of dimensions of the approximation function f (x) 
serving as a polynomial. Specifically, in this case, n = 5, 
and L = 19. 

If we define each matrix of the normal equation shown 
in Expression (55) as the following Expressions (56) through 
(58) , the normal equation is represented as the following 
Expression (59) . 
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. . - (57) 
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. . . (58) 



S MAT W MAT ~ P MAT 



. . . (59) 



As shown in Expression (57) , the respective components 
of the matrix Wma T are the features wi to be obtained. 



left side and the matrix Pmat of the right side are 
determined, the matrix VJma T (i.e., features w A ) may be 
calculated with the matrix solution. 

Specifically, as shown in Expression (56) , the 
respective components of the matrix Sma T may be calculated as 
long as the above integral components Si (1) are known. The 
integral components Si (1) are included in the integral 
component table supplied from the integral component 
calculation unit 2334, so the normal equation generating 
unit 2335 can calculate each component of the matrix Sma T 
using the integral component table. 



Accordingly, in Expression (59) , if the matrix Smat of the 
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Also, as shown in Expression (58) , the respective 
components of the matrix Pmat may be calculated as long as 
the integral components Si (1) and the input pixel values P 
(1) are known. The integral components S± (1) is the same 
5 as those included in the respective components of the matrix 
Smat, also the input pixel values P (1) are included in the 
input pixel value table supplied from the input pixel value 
acquiring unit 2333, so the normal equation generating unit 
2335 can calculate each component of the matrix Pmat using 
10 the integral component table and input pixel value table. 

Thus, the normal equation generating unit 233 5 
calculates each component of the matrix Sma T and matrix Pmat/ 
and outputs the calculated results (each component of the 
matrix Smat and matr ix Pmat) to the approximation function 
15 generating unit 233 6 as a normal equation table. 

Upon the normal equation table being output from the 
normal equation generating unit 2335, in step S2306, the 
approximation function generating unit 2336 calculates the 
features wi (i.e., the coefficients w ± of the approximation 
20 function f (x) serving as a one -dimensional polynomial) 

serving as the respective components of the matrix Wmat in 

the above Expression (59) based on the normal equation table. 

Specifically, the normal equation in the above 
Expression (59) can be transformed as the following 
25 Expression (60) . 
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W MAT " S MAT P MAT 

. . . (60) 

In Expression (60) , the respective components of the 
matrix Wmat in the left side are the features wi to be 
5 obtained. The respective components regarding the matrix 

Smat and matrix Pma T are included in the normal equation table 
supplied from the normal equation generating unit 2335. 
Accordingly, the approximation function generating unit 2336 
calculates the matrix Wma T by calculating the matrix in the 

10 right side of Expression (60) using the normal equation 

table, and outputs the calculated results (features wi) to 
the image generating unit 103. 

In step S2307, the approximation function generating 
unit 233 6 determines regarding whether or not the processing 

15 of all the pixels has been completed. 

In step S2307, in the event that determination is made 
that the processing of all the pixels has not been completed, 
the processing returns to step S2302, wherein the subsequent 
processing is repeatedly performed. That is to say, the 

20 pixels that have not become a pixel of interest are 
sequentially taken as a pixel of interest, and the 
processing in step S2302 through S23 07 is repeatedly 
performed . 

In the event that the processing of all the pixels has 
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been completed (in step S2307, in the event that 
determination is made that the processing of all the pixels 
has been completed) , the estimating processing of the actual 
world 1 ends . 

5 Note that the waveform of the approximation function 

f (x) generated with the coefficients (features) w± thus 
calculated becomes a waveform such as the approximation 
function f 3 (x) in Fig. 73 described above. 

Thus, with the one-dimensional approximating method, 

10 the features of the approximation function f (x) serving as a 
one -dimensional polynomial are calculated under the 
assumption, for example, that a waveform having the same 
form as the one-dimensional X cross-sectional waveform F (x) 
is continuous in the direction of continuity. Accordingly, 

15 with the one-dimensional approximating method, the features 
of the approximation function f (x) can be calculated with 
less amount of calculation processing than other function 
approximating methods. 

Next, description will be made regarding the second 

2 0 function approximating method with reference to Fig. 77 
through Fig. 83 . 

That is to say, the second function approximating 
method is a method wherein the light signal in the actual 
world 1 having continuity in the spatial direction 

25 represented with the gradient G F such as shown in Fig. 77 
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for example is regarded as a waveform F(x, y) on the X-Y 
plane (on the plane level in the X direction serving as one 
direction of the spatial directions, and in the Y direction 
orthogonal to the X direction), and the waveform F (x, y) is 
5 approximated with the approximation function f (x, y) serving 
as a two-dimensional polynomial, thereby estimating the 
waveform F(x, y) . Accordingly, hereafter, the second 
function approximating method is referred to as a two- 
dimensional polynomial approximating method. 

10 Note that in Fig. 77, the horizontal direction 

represents the X direction serving as one direction of the 
spatial directions, the upper right direction represents the 
Y direction serving as the other direction of the spatial 
directions, and the vertical direction represents the level 

15 of light respectively. G F represents the gradient as 
continuity in the spatial direction. 

Also, with description of the two-dimensional 
polynomial approximating method, let us say that the sensor 
2 is a CCD made up of the multiple detecting elements 2-1 

2 0 disposed on the plane thereof, such as shown in Fig. 78. 

With the example in Fig. 78, the direction in parallel 
with a predetermined side of the detecting elements 2-1 is 
taken as the X direction serving as one direction of the 
spatial directions, and the direction orthogonal to the X 

25 direction is taken as the Y direction serving as the other 
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direction of the spatial directions. The direction 
orthogonal to the X-Y plane is taken as the t direction 
serving as the temporal direction. 

Also, with the example in Fig. 78, the spatial shape of 
5 the respective detecting elements 2-1 of the sensor 2 is 
taken as a square of which one side is 1 in length. The 
shutter time (exposure time) of the sensor 2 is taken as 1. 

Further, with the example in Fig. 78, the center of one 
certain detecting element 2-1 of the sensor 2 is taken as 

10 the origin (the position in the X direction is x = 0, and 
the position in the Y direction is y = 0) in the spatial 
directions (X direction and Y direction) , and also the 
intermediate point - in- time of the exposure time is taken as 
the origin (the position in the t direction is t = 0) in the 

15 temporal direction (t direction) . 

In this case, the detecting element 2-1 of which the 
center is in the origin (x = 0, y = 0) in the spatial 
directions subjects the light signal function F(x, y, t) to 
integration with a range of -0.5 through 0.5 in the X 

20 direction, with a range of -0.5 through 0.5 in the Y 

direction, and with a range of -0.5 through 0.5 in the t 
direction, and outputs the integral value as the pixel value 
P. 

That is to say, the pixel value P output from the 
25 detecting element 2-1 of which the center is in the origin 
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in the spatial directions is represented with the following 
Expression (61) . 

P = /_ 0 . 5 /_ 05 /_ 05 F(x.y,t)dxdydt 

. . - (61) 

Similarly, the other detecting elements 2-1 output the 
pixel value P shown in Expression (61) by taking the center 
of the detecting element 2-1 to be processed as the origin 
in the spatial directions. 

Incidentally, as described above, the two-dimensional 
polynomial approximating method is a method wherein the 
light signal in the actual world 1 is handled as a waveform 
F(x, y) such as shown in Fig. 77 for example, and the two- 
dimensional waveform F(x, y) is approximated with the 
approximation function f (x, y) serving as a two-dimensional 
polynomial . 

First, description will be made regarding a method 
representing such the approximation function f (x, y) with a 
two-dimensional polynomial . 

As described above, the light signal in the actual 
world 1 is represented with the light signal function F(x, y, 
t) of which variables are the position on the three- 
dimensional space x, y, and z, and point -in- time t. This 
light signal function F(x, y, t) , i.e., a one-dimensional 
waveform projected in the X direction at an arbitrary 
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position y in the Y direction is referred to as an X cross- 
sectional waveform F (x) , here. 

When paying attention to this X cross-sectional 
waveform F (x) , in the event that the signal in the actual 
5 world 1 has continuity in a certain direction in the spatial 
directions, it can be conceived that a waveform having the 
same form as the X cross-sectional waveform F (x) continues 
in the continuity direction. For example, with the example 
in Fig. 77, a waveform having the same form as the X cross- 

10 sectional waveform F (x) continues in the direction of the 
gradient G F . In other words, it can be said that the 
waveform F(x, y) is formed by a waveform having the same 
form as the X cross-sectional waveform F (x) continuing in 
the direction of the gradient G F . 

15 Accordingly, the approximation function f (x, y) can be 

represented with a two-dimensional polynomial by considering 
that the waveform of the approximation function f (x, y) 
approximating the waveform F(x, y) is formed by a waveform 
having the same form as the approximation function f (x) 

2 0 approximating the X cross-sectional F (x) continuing. 

Description will be made in more detail regarding the 
representing method of the approximation function f (x, y) . 

For example, let us say that the light signal in the 
actual world 1 such as shown in Fig. 77 described above, 

25 i.e., a light signal having continuity in the spatial 
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direction represented with the gradient G F is detected by 
the sensor 2 (Fig. 78), and output as an input image (pixel 
value) . 

Further, let us say that as shown in Fig. 79, the data 
5 continuity detecting unit 101 (Fig. 3) subjects an input 
image region 2401 made up of 20 pixels (in the drawing, 20 
squares represented with dashed line) in total of 4 pixels 
in the X direction and also 5 pixels in the Y direction, of 
this input image, to the processing thereof, and outputs an 

10 angle 0 (angle 0 generated between the direction of data 

continuity represented with the gradient G f corresponding to 
the gradient G F , and the X direction) as one of the data 
continuity information . 

Note that with the input image region 2401, the 

15 horizontal direction in the drawing represents the X 

direction serving as one direction in the spatial directions, 
and the vertical direction in the drawing represents the Y 
direction serving as the other direction of the spatial 
directions . 

20 Also, in Fig. 79, an (x, y) coordinates system is set 

such that a pixel in the second pixel from the left, and 
also the third pixel from the bottom is taken as a pixel of 
interest, and the center of the pixel of interest is taken 
as the origin (0, 0) . A relative distance (hereafter, 

25 referred to as a cross-sectional direction distance) in the 
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X direction as to the straight line (straight line having 
the gradient G f representing the direction of data 
continuity) having an angle G passing through the origin (0, 
0) is described as x 1 . 

Further, in Fig. 79, the graph on the right side is a 
function wherein an X cross-sectional waveform F(x') is 
approximated, which represents an approximation function 
f (x') serving as an n-dimensional (n is an arbitrary 
integer) polynomial. Of the axes in the graph on the right 
side, the axis in the horizontal direction in the drawing 
represents a cross-sectional direction distance, and the 
axis in the vertical direction in the drawing represents 
pixel values . 

In this case, the approximation function f (x ! ) shown in 
Fig. 79 is an n-dimensional polynomial, so is represented as 
the following Expression (62) . 

n 

f(x') = W 0 +W 1 X'+W 2 X' + - • • +W n x' n = Z W|X M 

i=0 

. . . (62) 

Also, since the angle 0 is determined, the straight line 
having angle 0 passing through the origin (0, 0) is uniquely 
determined, and a position xi in the X direction of the 
straight line at an arbitrary position y in the Y direction 
is represented as the following Expression (63) . However, 
in Expression (63) , s represents cot 0 (= 1 ""□□□©) . 
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X| = sxy 

. . . (63) 

That is to say, as shown in Fig. 79, a point on the 
straight line corresponding to continuity of data 
represented with the gradient G f is represented with a 
coordinate value (x 1# y) . 

The cross-sectional direction distance x' is 
represented as the following Expression (64) using 
Expression (63) . 
X' = X~X| = X-S X y 

. . . (64) 

Accordingly, the approximation function f (x, y) at an 
arbitrary position (x, y) within the input image region 2401 
is represented as the following Expression (65) using 
Expression (62) and Expression (64) . 
n 

f (x, y) = Z Wj (x-sxy) 1 

i=0 

. . . (65) 

Note that in Expression (65) , wi represents coefficients 
of the approximation function f (x, y) . Note that the 
coefficients w± of the approximation function f including 
the approximation function f (x, y) can be evaluated as the 
features of the approximation function f. Accordingly, the 
coefficients Wi of the approximation function f are also 
referred to as the features wi of the approximation function 
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f . 

Thus, the approximation function f (x, y) having a two- 
dimensional waveform can be represented as the polynomial of 
Expression (65) as long as the angle 0 is known. 
5 Accordingly, if the actual world estimating unit 102 

can calculate the features Wi of Expression (65) , the actual 
world estimating unit 102 can estimate the waveform F(x, y) 
such as shown in Fig. 77. 

Consequently, hereafter, description will be made 

10 regarding a method for calculating the features wi of 
Expression (65) . 

That is to say, upon the approximation function f (x, y) 
represented with Expression (65) being subjected to 
integration with an integral range (integral range in the 

15 spatial direction) corresponding to a pixel (the detecting 
element 2-1 of the sensor 2 (Fig. 78)), the integral value 
becomes the estimated value regarding the pixel value of the 
pixel. It is the following Expression (66) that this is 
represented with an equation. Note that with the two- 

2 0 dimensional polynomial approximating method, the temporal 
direction t is regarded as a constant value, so Expression 
(66) is taken as an equation of which variables are the 
positions x and y in the spatial directions (X direction and 
Y direction) . 
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J«y+0. 5 px+O. 5 n ■ 
,-0.5 £-0.5 IWi(X-S*y>'+* 

. . . (66) 

In Expression (66) , P (x, y) represents the pixel value 
of a pixel of which the center position is in a position (x, 
5 y) (relative position (x, y) from the pixel of interest) of 
an input image from the sensor 2. Also, e represents a 
margin of error. 

Thus, with the two-dimensional polynomial approximating 
method, the relation between the input pixel value P (x, y) 

10 and the approximation function f (x, y) serving as a two- 
dimensional polynomial can be represented with Expression 
(66) , and accordingly, the actual world estimating unit 102 
can estimate the two-dimensional function F(x, y) (waveform 
F(x, y) wherein the light signal in the actual world 1 

15 having continuity in the spatial direction represented with 
the gradient G F (Fig. 77) is represented focusing attention 
on the spatial direction) by calculating the features Wi 
with, for example, the least square method or the like using 
Expression (66) (by generating the approximation function 

20 f (x, y) by substituting the calculated features w± for 
Expression (64) ) . 

Fig. 80 represents a configuration example of the 
actual world estimating unit 102 employing such a two- 
dimensional polynomial approximating method. 
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As shown in Fig. 80, the actual world estimating unit 
102 includes a conditions setting unit 2421, input image 
storage unit 2422, input pixel value acquiring unit 2423, 
integral component calculation unit 2424, normal equation 
5 generating unit 2425, and approximation function generating 
unit 2426. 

The conditions setting unit 2421 sets a pixel range 
(tap range) used for estimating the function F (x, y) 
corresponding to a pixel of interest, and the number of 
10 dimensions n of the approximation function f (x, y) . 

The input image storage unit 2422 temporarily stores an 
input image (pixel values) from the sensor 2. 

The input pixel value acquiring unit 2423 acquires, of 
the input images stored in the input image storage unit 2422, 
15 an input image region corresponding to the tap range set by 
the conditions setting unit 2421, and supplies this to the 
normal equation generating unit 2425 as an input pixel value 
table. .That is to say, the input pixel value table is a 
table in which the respective pixel values of pixels 
2 0 included in the input image region are described. Note that 
a specific example of the input pixel value table will be 
described later. 

Incidentally, as described above, the actual world 
estimating unit 102 employing the two-dimensional 
2 5 approximating method calculates the features wi of the 
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approximation function f (x, y) represented with the above 
Expression (65) by solving the above Expression (66) using 
the least square method. 

Expression (66) can be represented as the following 
Expression (71) by using the following Expression (70) 
obtained by the following Expressions (67) through (69) . 

,i+1 
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i+1 
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P(x.y) = I 



-(x+0. 5-sxy-O. 5s) i+2 - (x-0. 5-sxy+O. 5s) 1 




j +2 



+ (x-0. 5-sxy-O. 5s) i+2 }+e 



n 



= I WjSj (x-0. 5, x+0. 5, y-0. 5, y+0. 5) +e 



i=0 



- - . (71) 



In Expression (71), S± (x - 0.5, x + 0.5 



, Y 



- 0.5, y + 



0.5) represents the integral components of i -dimensional 
terms. That is to say, the integral components Si (x - 0.5, 
x + 0.5, y - 0.5, y + 0.5) are as shown in the following 
Expression (72) . 



Sj (x-0. 5, x+0. 5, y-0. 5, y+0. 5) = 

(x+0.5-sxy+0.5s) i+2 -(x+0.5-sxy-0. 5s) i+2 -(x-0. 5-sxy+O. 5s) i+2 +(x-0. 5-sxy-Q. 5s) 1 

s(i+1)(i+2) 



The integral component calculation unit 2424 calculates 
the integral components S± (x- 0.5, x+0. 5, y- 0.5, y+ 
0.5). 

Specifically, the integral components Si (x - 0.5, x + 
0.5, y - 0.5, y + 0.5) shown in Expression (72) can be 
calculated as long as the relative pixel positions (x, y) , 
the variable s and i of i -dimensional terms in the above 
Expression (65) are known. Of these, the relative pixel 
positions (x, y) are determined with a pixel of interest, 



. . . (72) 
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and a tap range, the variable s is cot 0, which is 
determined with the angle 0, and the range of i is 
determined with the number of dimensions n respectively. 
Accordingly, the integral component calculation unit 
5 2424 calculates the integral components Si (x- 0.5, x+0.5, 
y- 0.5, y + 0.5) based on the tap range and the number of 
dimensions set by the conditions setting unit 2421, and the 
angle 0 of the data continuity information output from the 
data continuity detecting unit 101, and supplies the 

10 calculated results to the normal equation generating unit 
2425 as an integral component table. 

The normal equation generating unit 242 5 generates a 
normal equation in the case of obtaining the above 
Expression (66), i.e., Expression (71) by the least square 

15 method using the input pixel value table supplied from the 
input pixel value acquiring unit 2423, and the integral 
component table supplied from the integral component 
calculation unit 2424, and outputs this to the approximation 
function generating unit 2426 as a normal equation table. 

20 Note that a specific example of a normal equation will be 
described later. 

The approximation function generating unit 2426 
calculates the respective features w± of the above 
Expression (66) (i.e., the coefficients wi of the 

25 approximation function f (x, y) serving as a two-dimensional 
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polynomial) by solving the normal equation included in the 
normal equation table supplied from the normal equation 
generating unit 2425 using the matrix solution, and output 
these to the image generating unit 103. 
5 Next, description will be made regarding the actual 

world estimating processing (processing in step S102 in Fig. 
29) to which the two-dimensional polynomial approximating 
method is applied, with reference to the flowchart in Fig. 
81. 

10 For example, let us say that the light signal in the 

actual world 1 having continuity in the spatial direction 
represented with the gradient G F has been detected by the 
sensor 2 (Fig. 78) , and has been stored in the input image 
storage unit 2422 as an input image corresponding to one 

15 frame. Also, let us say that the data continuity detecting 
unit 101 has subjected the region 2401 shown in Fig. 79 
described above of the input image to processing in the 
continuity detecting processing in step S101 (Fig. 29) , and 
has output the angle 6 as data continuity information. 

20 In this case, in step S2401, the conditions setting 

unit 2421 sets conditions (a tap range and the number of 
dimensions) . 

For example, let us say that a tap range 2441 shown in 
Fig. 82 has been set, and also 5 has been set as the number 
25 of dimensions. 
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Fig. 82 is a diagram for describing an example of a tap 
range. In Fig. 82, the X direction and Y direction 
represent the X direction and Y direction of the sensor 2 
(Fig. 78) . Also, the tap range 2441 represents a pixel 
5 group made up of 20 pixels (20 squares in the drawing) in 
total of 4 pixels in the X direction and also 5 pixels in 
the Y direction. 

Further, as shown in Fig. 82, let us say that a pixel 
of interest has been set to a pixel, which is the second 

10 pixel from the left and also the third pixel from the bottom 
in the drawing, of the tap range 2441. Also, let us say 
that each pixel is denoted with a number 1 such as shown in 
Fig. 82 (1 is any integer value of 0 through 19) according 
to the relative pixel positions (x, y) from the pixel of 

15 interest (a coordinate value of a pixel -of - interest 

coordinates system wherein the center (0, 0) of the pixel of 
interest is taken as the origin) . 

Now, description will return to Fig. 81, wherein in 
step S2402, the conditions setting unit 2421 sets a pixel of 

20 interest. 

In step S2403, the input pixel value acquiring unit 
2423 acquires an input pixel value based on the condition 
(tap range) set by the conditions setting unit 2421, and 
generates an input pixel value table. That is to say, in 

25 this case, the input pixel value acquiring unit 2423 
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acquires the input image region 2401 (Fig. 79), generates a 
table made up of 2 0 input pixel values P (1) as an input 
pixel value table. 

Note that in this case, the relation between the input 
5 pixel values P (1) and the above input pixel values P (x, y) 
is a relation shown in the following Expression (73) . 
However, in Expression (73) , the left side represents the 
input pixel values P (1) , and the right side represents the 
input pixel values P (x, y) . 
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In step S2404, the integral component calculation unit 
2424 calculates integral components based on the conditions 
(a tap range and the number of dimensions) set by the 
conditions setting unit 2421, and the data continuity 
information (angle 0) supplied from the data continuity 
detecting unit 101, and generates an integral component 
table . 

In this case, as described above, the input pixel 
values are not P (x, y) but P (1) , and are acquired as the 
value of a pixel number 1, so the integral component 
calculation unit 2424 calculates the integral components Si 
(x - 0.5, x+ 0.5, y - 0.5, y + 0.5) in the above Expression 
(72) as a function of 1 such as the integral components Si 
(1) shown in the left side of the following Expression (74) . 

Sj ( I ) = Sj (x-0. 5, x+0. 5, y-0. 5, y+0. 5) 

. . . (74) 

Specifically, in this case, the integral components Si 
(1) shown in the following Expression (75) are calculated. 
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10 



. . . (75) 

Note that in Expression (75) , the left side represents 
the integral components Si (1), and the right side 
represents the integral components S± (x - 0.5, x+ 0.5, y - 
0.5, y + 0.5) . That is to say, in this case, i is 0 through 
5, and accordingly, the 12 0 Si (1) in total of the 2 0 S 0 (1) , 
20 Si (1) , 20 S 2 (1) , 20 S 3 (1) , 20 S 4 (1) , and 20 S 5 (1) are 
calculated . 

More specifically, first the integral component 



calculation unit 2424 calculates cot 0 corresponding to the 
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angle 0 supplied from the data continuity detecting unit 101, 
and takes the calculated result as a variable s. Next, the 
integral component calculation unit 2424 calculates each of 
the 2 0 integral components S± (x - 0.5, x + 0.5, y- 0.5, y 
5 +0.5) shown in the right side of Expression (74) regarding 
each of i = 0 through 5 using the calculated variable s. 
That is to say, the 12 0 integral components Si (x - 0.5, x + 
0.5, y - 0.5, y + 0.5) are calculated. For computing the 
integration component Si (x- 0.5, x + 0.5, y- 0.5, y + 

10 0.5), the above-described Expression (72) is used. Also the 
integration component computing unit 2424 converts each of 
the 12 0 computed integration components Si (x- 0.5, x+0.5, 
y - 0.5, y + 0.5) into the corresponding Si (1) respectively, 
according to Expression (75) , and generates an integration 

15 component table containing the 12 0 converted Si (1) . 

Note that the sequence of the processing in step S2403 
and the processing in step S2404 is not restricted to the 
example in Fig. 81, the processing in step S2404 may be 
executed first, or the processing in step S2403 and the 

20 processing in step S2404 may be executed simultaneously. 

Next, in step S2405, the normal equation generating 
unit 2425 generates a normal equation table based on the 
input pixel value table generated by the input pixel value 
acquiring unit 2423 at the processing in step S2403, and the 

25 integral component table generated by the integral component 
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calculation unit 2424 at the processing in step S2404. 

Specifically, in this case, the features wi are 
calculated with the least square method using the above 
Expression (71) (however, in Expression (70) , the S± (1) 
into which the integral components Si (x-0.5,x+0.5, y - 
0.5, y + 0.5) are converted using Expression (74) is used), 
so a normal equation corresponding to this is represented as 
the following Expression (76) . 
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10 ... (76) 

Note that in Expression (76) , L represents the maximum 
value of the pixel number 1 in the tap range. n represents 
the number of dimensions of the approximation function f (x) 
serving as a polynomial. Specifically, in this case, n = 5, 

15 and L = 19. 

If we define each matrix of the normal equation shown 
in Expression (76) as the following Expressions (77) through 
(79) , the normal equation is represented as the following 
Expression (8 0) . 
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As shown in Expression (78) , the respective components 
of the matrix Wma T are the features w± to be obtained. 
Accordingly, in Expression (80) , if the matrix Sma T of the 
5 left side and the matrix Pma T of the right side are 

determined, the matrix Wma T may be calculated with the matrix 
solution. 

Specifically, as shown in Expression (77) , the 
. respective components of the matrix Sma T may be calculated 

10 with the above integral components S± (1) . That is to say, 
the integral components Si (1) are included in the integral 
component table supplied from the integral component 
calculation unit 2424, so the normal equation generating 
unit 242 5 can calculate each component of the matrix Smat 

15 using the integral component table. 

Also, as shown in Expression (79) , the respective 
components of the matrix Pmat may be calculated with the 
integral components Si (1) and the input pixel values P (1) . 
That is to say, the integral components Si (1) is the same 

2 0 as those included in the respective components of the matrix 
Smat, also the input pixel values P (1) are included in the 
input pixel value table supplied from the input pixel value 
acquiring unit 2423, so the normal equation generating unit 
2425 can calculate each component of the matrix P^t using 

2 5 the integral component table and input pixel value table. 
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Thus, the normal equation generating unit 2425 
calculates each component of the matrix Smat and matrix Pmat, 
and outputs the calculated results (each component of the 
matrix Smat and matrix Pmat) to the approximation function 
5 generating unit 2426 as a normal equation table. 

Upon the normal equation table being output from the 
normal equation generating unit 2425, in step S2406, the 
approximation function generating unit 2426 calculates the 
features wi (i.e., the coefficients wi of the approximation 
10 function f (x, y) serving as a two-dimensional polynomial) 
serving as the respective components of the matrix Wmat in 
the above Expression (80) based on the normal equation table. 

Specifically, the normal equation in the above 
Expression (80) can be transformed as the following 
15 Expression (81) . 

W MAT = S MAT P MAT 

. . . (81) 

In Expression (81) , the respective components of the 
matrix Wmat in the left side are the features w± to be 
2 0 obtained. The respective components regarding the matrix 

Smat and matrix Pmat are included in the normal equation table 
supplied from the normal equation generating unit 2425. 
Accordingly, the approximation function generating unit 2426 
calculates the matrix Wmat by calculating the matrix in the 



1 
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right side of Expression (81) using the normal equation 
table, and outputs the calculated results (features wi) to 
the image generating unit 103. 

In step S2407, the approximation function generating 
5 unit 242 6 determines regarding whether or not the processing 
of all the pixels has been completed. 

In step S24 07, in the event that determination is made 
that the processing of all the pixels has not been completed, 
the processing returns to step S2402, wherein the subsequent 
10 processing is repeatedly performed. That is to say, the 
pixels that have not become a pixel of interest are 
sequentially taken as a pixel of interest, and the 
processing in step S2402 through S2407 is repeatedly 
performed. 

15 In the event that the processing of all the pixels has 

been completed (in step S2407, in the event that 
determination is made that the processing of all the pixels 
has been completed) , the estimating processing of the actual 
world 1 ends. 

20 As description of the two-dimensional polynomial 

approximating method, an example for calculating the 
coefficients (features) Wi of the approximation function f (x, 
y) corresponding to the spatial directions (X direction and 
Y direction) has been employed, but the two-dimensional 

2 5 polynomial approximating method can be applied to the 
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temporal and spatial directions (X direction and t direction, 
or Y direction and t direction) as well. 

That is to say, the above example is an example in the 
case of the light signal in the actual world 1 having 
5 continuity in the spatial direction represented with the 
gradient G F (Fig. 77), and accordingly, the equation 
including two-dimensional integration in the spatial 
directions (X direction and Y direction) , such as shown in 
the above Expression (66) . However, the concept regarding 
10 two-dimensional integration can be applied not only to the 
spatial direction but also to the temporal and spatial 
directions (X direction and t direction, or Y direction and 
t direction) . 

In other words, with the two-dimensional polynomial 
15 approximating method, even in the case in which the light 

signal function F (x, y, t) , which needs to be estimated, has 
not only continuity in the spatial direction but also 
continuity in the temporal and spatial directions (however, 
X direction and t direction, or Y direction and t direction) , 
2 0 this can be approximated with a two-dimensional 
approximation function f. 

Specifically, for example, in the event that there is 
an object moving horizontally in the X direction at uniform 
velocity, the direction of movement of the object is 
25 represented with like a gradient V F in the X-t plane such as 
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shown in Fig. 83. In other words, it can be said that the 
gradient V F represents the direction of continuity in the 
temporal and spatial directions in the X-t plane. 
Accordingly, the data continuity detecting unit 101 can 
5 output movement 0 such as shown in Fig. 8 3 (strictly 

speaking, though not shown in the drawing, movement 0 is an 
angle generated by the direction of data continuity 
represented with the gradient V f corresponding to the 
gradient V F and the X direction in the spatial direction) as 

10 data continuity information corresponding to the gradient V F 
representing continuity in the temporal and spatial 
directions in the X-t plane as well as the above angle 0 
(data continuity information corresponding to continuity in 
the spatial directions represented with the gradient G F in 

15 the X-Y plane) . 

Accordingly, the actual world estimating unit 102 
employing the two-dimensional polynomial approximating 
method can calculate the coefficients (features) wi of an 
approximation function f (x, t) in the same method as the 

2 0 above method by employing the movement 0 instead of the 

angle 0. However, in this case, the equation to be employed 
is not the above Expression (66) but the following 
Expression (82) . 

J-t+0. 5 r x+0. 5 n 
t-0.5 Jx-0.5 Z*i(x-8xt)'dxdt+e 
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Note that in Expression (82) , s is cot 0 (however, 0 is 
movement) . 

Also, an approximation function f (y, t) focusing 
5 attention on the spatial direction Y instead of the spatial 
direction X can be handled in the same way as the above 
approximation function f (x, t) . 

Thus, the two-dimensional polynomial approximating 
method takes not one -dimensional but two-dimensional 
10 integration effects into consideration, so can estimate the 
light signals in the actual world 1 more accurately than the 
one -dimensional approximating method. 

Next, description will be made regarding the third 
function approximating method with reference to Fig. 84 
15 through Fig. 88. 

That is to say, the third function approximating method 
is a method for estimating the light signal function F (x, y, 
t) by approximating the light signal function F(x, y, t) 
with the approximation function f (x, y, t) focusing 
20 attention on that the light signal in the actual world 1 
having continuity in a predetermined direction of the 
temporal and spatial directions is represented with the 
light signal function F(x, y, t), for example. Accordingly, 
hereafter, the third function approximating method is 
25 referred to as a three-dimensional function approximating 
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method. 

Also, with description of the three-dimensional 
function approximating method, let us say that the sensor 2 
is a CCD made up of the multiple detecting elements 2-1 
5 disposed on the plane thereof, such as shown in Fig. 84. 

With the example in Fig. 84, the direction in parallel 
with a predetermined side of the detecting elements 2-1 is 
taken as the X direction serving as one direction of the 
spatial directions, and the direction orthogonal to the X 
10 direction is taken as the Y direction serving as the other 
direction of the spatial directions. The direction 
orthogonal to the X-Y plane is taken as the t direction 
serving as the temporal direction. 

Also, with the example in Fig. 84, the spatial shape of 
15 the respective detecting elements 2-1 of the sensor 2 is 
taken as a square of which one side is 1 in length. The 
shutter time (exposure time) of the sensor 2 is taken as 1. 

Further, with the example in Fig. 84, the center of one 
certain detecting element 2-1 of the sensor 2 is taken as 
20 the origin (the position in the X direction is x = 0, and 
the position in the Y direction is y = 0) in the spatial 
directions (X direction and Y direction) , and also the 
intermediate point -in- time of the exposure time is taken as 
the origin (the position in the t direction is t = 0) in the 
25 temporal direction (t direction) . 
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In this case, the detecting element 2-1 of which the 
center is in the origin (x = 0, y = 0) in the spatial 
directions subjects the light signal function F(x, y, t) to 
integration with a range of -0.5 through 0.5 in the X 
direction, with a range of -0.5 through 0.5 in the Y 
direction, and with a range of -0.5 through 0.5 in the t 
direction, and outputs the integral value as the pixel value 
P. 

That is to say, the pixel value P output from the 
detecting element 2-1 of which the center is in the origin 
in the spatial directions is represented with the following 
Expression (83) . 

/-MX 5 /•+(). 5 r+0. 5 
-0.5 /-o.5 /-0.5 F(x,y,t)dxdydt 

. . . (83) 

Similarly, the other detecting elements 2-1 output the 
pixel value P shown in Expression (83) by taking the center 
of the detecting element 2-1 to be processed as the origin 
in the spatial directions. 

Incidentally, as described above, with the three- 
dimensional function approximating method, the light signal 
function F(x, y, t) is approximated to the three-dimensional 
approximation function f (x, y, t) . 

Specifically, for example, the approximation function 
f (x, y, t) is taken as a function having N variables 
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(features) , a relational expression between the input pixel 
values P (x, y, t) corresponding to Expression (83) and the 
approximation function f (x, y, t) is defined. Thus, in the 
event that M input pixel values P (x, y, t) more than N are 
5 acquired, N variables (features) can be calculated from the 
defined relational expression. That is to say, the actual 
world estimating unit 102 can estimate the light signal 
function F(x, y, t) by acquiring M input pixel values P (x, 
y, t) , and calculating N variables (features) . 

10 In this case, the actual world estimating unit 102 

extracts (acquires) M input images P (x, y, t) , of the 
entire input image by using continuity of data included in 
an input image (input pixel values) from the sensor 2 as a 
constraint (i.e., using data continuity information as to an 

15 input image to be output from the data continuity detecting 
unit 101). As a result, the approximation function f (x, y, 
t) is constrained by continuity of data. 

For example , as shown in Fig . 85 , in the event that the 
light signal function F(x, y, t) corresponding to an input 

20 image has continuity in the spatial direction represented 

with the gradient G F , the data continuity detecting unit 101 
results in outputting the angle 0 (the angle 9 generated 
between the direction of continuity of data represented with 
the gradient G f (not shown) corresponding to the gradient G F , 

25 and the X direction) as data continuity information as to 
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the input image . 

In this case, let us say that a one-dimensional 
waveform wherein the light signal function F(x, y, t) is 
projected in the X direction (such a waveform is referred to 
5 as an X cross-sectional waveform here) has the same form 
even in the event of projection in any position in the Y 
direction. 

That is to say, let us say that there is an X cross- 
sectional waveform having the same form, which is a two- 

10 dimensional (spatial directional) waveform continuous in the 
direction of continuity (angle 9 direction as to the X 
direction) , and a three-dimensional waveform wherein such a 
two-dimensional waveform continues in the temporal direction 
t, is approximated with the approximation function f (x, y, 

15 t) . 

In other words, an X cross-sectional waveform, which is 
shifted by a position y in the Y direction from the center 
of the pixel of interest, becomes a waveform wherein the X 
cross-sectional waveform passing through the center of the 

20 pixel of interest is moved (shifted) by a predetermined 
amount (amount varies according to the angle 9) in the X 
direction. Note that hereafter, such an amount is referred 
to as a shift amount. 

This shift amount can be calculated as follows. 

25 That is to say, the gradient V f (for example, gradient 
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V f representing the direction of data continuity 
corresponding to the gradient V F in Fig. 85) and angle 0 are 
represented as the following Expression (84) . 



G f = tan G = 




. . . (84) 

Note that in Expression (84) , dx represents the amount 
of fine movement in the X direction, and dy represents the 
amount of fine movement in the Y direction as to the dx. 

Accordingly, if the shift amount as to the X direction 
is described as C x (y) , this is represented as the following 
Expression (85) . 




. . . (85) 

If the shift amount C x (y) is thus defined, a relational 
expression between the input pixel values P (x, y, t) 
corresponding to Expression (83) and the approximation 
function f (x, y, t) is represented as the following 
Expression (86) . 

p(x, y ,t) = f te f ye r Xef(x -y- t)dxd y dt+e 

L s *s *s 

. . . (86) 

In Expression (86) , e represents a margin of error. t s 
represents an integration start position in the t direction, 
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and t e represents an integration end position in the t 
direction. In the same way, y s represents an integration 
start position in the Y direction, and y e represents an 
integration end position in the Y direction. Also, x s 
represents an integration start position in the X direction, 
and x e represents an integration end position in the X 
direction. However, the respective specific integral ranges 
are as shown in the following Expression (87) . 

t s = t-0. 5 
t e = t+0.5 

y s = y-o. 5 

y e = y+0. 5 

x s = x-C x (y)-0. 5 

x e = x-C x (y)+0. 5 

. . . (87) 

As shown in Expression (87) , it can be represented that 
an X cross-sectional waveform having the same form continues 
in the direction of continuity (angle 9 direction as to the 
X direction) by shifting an integral range in the X 
direction as to a pixel positioned distant from the pixel of 
interest by (x, y) in the spatial direction by the shift 
amount C x (y) . 

Thus, with the three-dimensional approximating method, 
the relation between the pixel values P (x, y, t) and the 
three-dimensional approximation function f (x, y, t) can be 
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represented with Expression (86) (Expression (87) for the 
integral range) , and accordingly, the light signal function 
F(x, y, t) (for example, a light signal having continuity in 
the spatial direction represented with the gradient V F such 
as shown in Fig. 85) can be estimated by calculating the N 
features of the approximation function f (x, y, t) , for 
example, with the least square method using Expression (86) 
and Expression (87) . 

Note that in the event that a light signal represented 
with the light signal function F(x, y, t) has continuity in 
the spatial direction represented with the gradient V F such 
as shown in Fig. 85, the light signal function F(x, y, t) 
may be approximated as follows. 

That is to say, let us say that a one -dimensional 
waveform wherein the light signal function F (x, y, t) is 
projected in the Y direction (hereafter, such a waveform is 
referred to as a Y cross -sectional waveform) has the same 
form even in the event of projection in any position in the 
X direction. 

In other words, let us say that there is a two- 
dimensional (spatial directional) waveform wherein a Y 
cross -sectional waveform having the same form continues in 
the direction of continuity (angle 0 direction as to in the 
X direction) , and a three-dimensional waveform wherein such 
a two-dimensional waveform continues in the temporal 
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direction t is approximated with the approximation function 
f (x, y, t) . 

Accordingly, the Y cross-sectional waveform, which is 
shifted by x in the X direction from the center of the pixel 
of interest, becomes a waveform wherein the Y cross- 
sectional waveform passing through the center of the pixel 
of interest is moved by a predetermined shift amount (shift 
amount changing according to the angle 0) in the Y direction. 

This shift amount can be calculated as follows. 

That is to say, the gradient G F is represented as the 
above Expression (84), so if the shift amount as to the Y 
direction is described as C y (x) , this is represented as the 
following Expression (88) . 

C y (x) = G f x x 

. . . (88) 

If the shift amount C x (y) is thus defined, a relational 
expression between the input pixel values P (x, y, t) 
corresponding to Expression (83) and the approximation 
function f (x, y, t) is represented as the above Expression 
(86) , as with when the shift amount C x (y) is defined. 

However, in this case, the respective specific integral 
ranges are as shown in the following Expression (89) . 
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t s = t-0. 5 
t e = t+0. 5 

y s = y-c y (x)-o. 5 
y e = y-c y (x)+o. 5 

x s = x-0. 5 
x e = x+0. 5 

. . . (89) 

As shown in Expression (89) (and the above Expression 
(86) ) , it can be represented that a Y cross-sectional 
5 waveform having the same form continues in the direction of 
continuity (angle G direction as to the X direction) by 
shifting an integral range in the Y direction as to a pixel 
positioned distant from the pixel of interest by (x, y) , by 
the shift amount C x (y) . 

10 Thus, with the three-dimensional approximating method, 

the integral range of the right side of the above Expression 
(86) can be set to not only Expression (87) but also 
Expression (89) , and accordingly, the light signal function 
F(x, y, t) (light signal in the actual world 1 having 

15 continuity in the spatial direction represented with the 

gradient G F ) can be estimated by calculating the n features 
of the approximation function f (x, y, t) with, for example, 
the least square method or the like using Expression (86) in 
which Expression (89) is employed as an integral range. 

20 Thus, Expression (87) and Expression (89) , which 
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represent an integral range, represent essentially the same 
with only a difference regarding whether perimeter pixels 
are shifted in the X direction (in the case of Expression 
(87)) or shifted in the Y direction (in the case of 
5 Expression (89) ) in response to the direction of continuity. 
However, in response to the direction of continuity 
(gradient G F ) , there is a difference regarding whether the 
light signal function F(x, y, t) is regarded as a group of X 
cross-sectional waveforms, or is regarded as a group of Y 
10 cross-sectional waveforms. That is to say, in the event 

that the direction of continuity is close to the Y direction, 
the light signal function F(x, y, t) is preferably regarded 
as a group of X cross-sectional waveforms. On the other 
hand, in the event that the direction of continuity is close 
15 to the X direction, the light signal function F(x, y, t) is 
preferably regarded as a group of Y cross - sectional 
waveforms . 

Accordingly, it is preferable that the actual world 
estimating unit 102 prepares both Expression (87) and 
2 0 Expression (89) as an integral range, and selects any one of 
Expression (87) and Expression (89) as the integral range of 
the right side of the appropriate Expression (86) in 
response to the direction of continuity. 

Description has been made regarding the three- 
2 5 dimensional approximation method in the case in which the 
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light signal function F(x, y, t) has continuity (for example, 
continuity in the spatial direction represented with the 
gradient G F in Fig. 85) in the spatial directions (X 
direction and Y direction) , but the three-dimensional 
5 approximation method can be applied to the case in which the 
light signal function F(x, y, t) has continuity (continuity 
represented with the gradient V F ) in the temporal and 
spatial directions (X direction, Y direction, and t 
direction), as shown in Fig. 86. 

10 That is to say, in Fig. 86, a light signal function 

corresponding to a frame #N - 1 is taken as F (x, y, #N - 1) , 
a light signal function corresponding to a frame #N is taken 
as F (x, y, #N) , and a light signal function corresponding 
to a frame #N + 1 is taken as F (x, y, #N + 1) . 

15 Note that in Fig. 86, the horizontal direction is taken 

as the X direction serving as one direction of the spatial 
directions, the upper right diagonal direction is taken as 
the Y direction serving as the other direction of the 
spatial directions, and also the vertical direction is taken 

2 0 as the t direction serving as the temporal direction in the 
drawing . 

Also, the frame #N - 1 is a frame temporally prior to 
the frame #N, the frame #N + 1 is a frame temporally 
following the frame #N. That is to say, the frame #N - 1 , 
2 5 frame #N, and frame #N + 1 are displayed in the sequence of 
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the frame #N - 1 , frame #N, and frame #N + 1 . 

With the example in Fig. 86, a cross-sectional light 
level along the direction shown with the gradient V F (upper 
right inner direction from lower left near side in the 
5 drawing) is regarded as generally constant. Accordingly, 
with the example in Fig. 86, it can be said that the light 
signal function F(x, y, t) has continuity in the temporal 
and spatial directions represented with the gradient V F . 

In this case, in the event that a function C (x, y, t) 

10 representing continuity in the temporal and spatial 

directions is defined, and also the integral range of the 
above Expression (86) is defined with the defined function C 
(x, y, t) , N features of the approximation function f(x, y, 
t) can be calculated as with the above Expression (87) and 

15 Expression (89) . 

The function C (x, y, t) is not restricted to a 
particular function as long as this is a function 
representing the direction of continuity. However, 
hereafter, let us say that linear continuity is employed, 

2 0 and C x (t) and C y (t) corresponding to the shift amount C x 
(y) (Expression (85) ) and shift amount C y (x) (Expression 
(87)), which are functions representing continuity in the 
spatial direction described above, are defined as a function 
C (x, y, t) corresponding thereto as follows. 

25 That is to say, if the gradient as continuity of data 



10 
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in the temporal and spatial directions corresponding to the 
gradient G f representing continuity of data in the above 
spatial direction is taken as V f/ and if this gradient V f is 
divided into the gradient in the X direction (hereafter, 
referred to as V fx ) and the gradient in the Y direction 
(hereafter, referred to as V fy ) , the gradient V fx is 
represented with the following Expression (90) , and the 
gradient V fy is represented with the following Expression 
(91) , respectively. 

Vfx dt 

. . . (90) 

Vf = 

. . . (91) 

In this case, the function C x (t) is represented as the 
15 following Expression (92) using the gradient V fx shown in 
Expression (90) . 

C x (t) =V fx xt 

■ • • (92) 

Similarly, the function C y (t) is represented as the 
2 0 following Expression (93) using the gradient V fy shown in 
Expression (91) . 

Cy(t) =V fy Xt 

. . . (93) 
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Thus, upon the function C x (t) and function C y (t) , 
which represent continuity 2511 in the temporal and spatial 
directions, being defined, the integral range of Expression 
(86) is represented as the following Expression (94) . 

t s = t-0. 5 
t e = t+0. 5 
y s = y-C y (t)-0.5 

y e = y-c y (t)+o. 5 

x s = x-C x (t)-0. 5 
5 x e = x-C x (t) +0. 5 

. . . (94) 

Thus, with the three-dimensional approximating method, 
the relation between the pixel values P (x, y, t) and the 
three-dimensional approximation function f (x, y, t) can be 

10 represented with Expression (86) , and accordingly, the light 
signal function F(x, y, t) (light signal in the actual world 
1 having continuity in a predetermined direction of the 
temporal and spatial directions) can be estimated by 
calculating the n + 1 features of the approximation function 

15 f (x, y, t) with, for example, the least square method or the 
like using Expression (94) as the integral range of the 
right side of Expression (86) . 

Fig. 87 represents a configuration example of the 
actual world estimating unit 102 employing such a three- 

2 0 dimensional approximating method. 
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Note that the approximation function f (x, y, t) (in 
reality, the features (coefficients) thereof) calculated by 
the actual world estimating unit 102 employing the three- 
dimensional approximating method is not restricted to a 
5 particular function, but an n (n = N - 1) -dimensional 
polynomial is employed in the following description. 

As shown in Fig. 87, the actual world estimating unit 
102 includes a conditions setting unit 2521, input image 
storage unit 2522, input pixel value acquiring unit 2523, 
10 integral component calculation unit 2524, normal equation 

generating unit 2525, and approximation function generating 
unit 2526. 

The conditions setting unit 2521 sets a pixel range 
(tap range) used for estimating the light signal function 
15 F(x, y, t) corresponding to a pixel of interest, and the 

number of dimensions n of the approximation function f (x, y, 
t) . 

The input image storage unit 2 522 temporarily stores an 
input image (pixel values) from the sensor 2. 

20 The input pixel acquiring unit 2523 acquires, of the 

input images stored in the input image storage unit 2522, an 
input image region corresponding to the tap range set by the 
conditions setting unit 2521, and supplies this to the 
normal equation generating unit 2525 as an input pixel value 

25 table. That is to say, the input pixel value table is a 
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table in which the respective pixel values of pixels 
included in the input image region are described. 

Incidentally, as described above, the actual world 
estimating unit 102 employing the three-dimensional 
5 approximating method calculates the N features (in this case, 
coefficient of each dimension) of the approximation function 
f (x, y, t) with the least square method using the above 
Expression (86) (however, Expression (87) , Expression (90) , 
or Expression (94) for the integral range) . 
10 The right side of Expression (86) can be represented as 

the following Expression (95) by calculating the integration 
thereof . 

n 

P(x, y, t) = I WjSj (x s , x e , y s , y e> t s , t e ) +e 

i=0 

. . . (95) 

15 In Expression (95) , wi represents the coefficients 

(features) of the i-dimensional term, and also Si (x s , x e , y s , 
y e , t s , t e ) represents the integral components of the i- 
dimensional term. However, x s represents an integral range 
start position in the X direction, x e represents an integral 

20 range end position in the X direction, y s represents an 
integral range start position in the Y direction, y e 
represents an integral range end position in the Y direction, 
t s represents an integral range start position in the t 
direction, t e represents an integral range end position in 
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the t direction, respectively. 

The integral component calculation unit 2524 calculates 
the integral components Si (x S/ x e , y S / Ye/ t s , t e ) . 

That is to say, the integral component calculation unit 
5 2 524 calculates the integral components Si (x S/ x e/ y S / Ye, ts, 
t e ) based on the tap- range and the number of dimensions set 
by the conditions setting unit 2521, and the angle or 
movement (as the integral range, angle in the case of using 
the above Expression (87) or Expression (90) , and movement 

i 

10 in the case of using the above Expression (94) ) of the data 
continuity information output from the data continuity 
detecting unit 101, and supplies the calculated results to 
the normal equation generating unit 2 52 5 as an integral 
component table . 

15 The normal equation generating unit 2525 generates a 

normal equation in the case of obtaining the above 
Expression (95) with the least square method using the input 
pixel value table supplied from the input pixel value 
acquiring unit 2523, and the integral component table 

2 0 supplied from the integral component calculation unit 2524, 
and outputs this to the approximation function generating 
unit 2 52 6 as a normal equation table. An example of a 
normal equation will be described later. 

The approximation function generating unit 2 52 6 

25 calculates the respective features Wi (in this case, the 
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coefficients wi of the approximation function f (x, y, t) 
serving as a three-dimensional polynomial) by solving the 
normal equation included in the normal equation table 
supplied from the normal equation generating unit 2525 with 
5 the matrix solution, and output these to the image 
generating unit 103 . 

Next, description will be made regarding the actual 
world estimating processing (processing in step S102 in Fig. 
29) to which the three-dimensional approximating method is 
10 applied, with reference to the flowchart in Fig. 88. 

First, in step S2501, the conditions setting unit 2521 
sets conditions (a tap range and the number of dimensions) . 

For example, let us say that a tap range made up of L J 

s 

pixels has been set. Also, let us say that a predetermined 
15 number 1 (1 is any one of integer values 0 through L - 1) is 

appended to each of the pixels. 

Next, in step S2502, the conditions setting unit 2521 

sets a pixel of interest . 

In step S2503, the input pixel value acquiring unit 
20 2523 acquires an input pixel value based on the condition 

(tap range) set by the conditions setting unit 2521, and 

generates an input pixel value table. In this case, a table 

made up of L input pixel values P (x, y, t) is generated. 

Here, let us say that each of the L input pixel values P (x, 
25 y, t) is described as P (1) serving as a function of the 
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number 1 of the pixel thereof. That is to say, the input 
pixel value table becomes a table including LP (1) . 

In step S2504, the integral component calculation unit 
2524 calculates integral components based on the conditions 
5 (a tap range and the number of dimensions) set by the 
conditions setting unit 2521, and the data continuity 
information (angle or movement) supplied from the data 
continuity detecting unit 101, and generates an integral 
component table. 

10 However, in this case, as described above, the input 

pixel values are not P (x, y, t) but P (1), and are acquired 
as the value of a pixel number 1, so the integral component 
calculation unit 2524 results in calculating the integral 
components Si (x s , x e , y S / y e / t s , t e ) in the above Expression 

15 (95) as a function of 1 such as the integral components Si 

(1) . That is to say, the integral component table becomes a- 

table including L x i Si (1) . 

Note that the sequence of the processing in step S2503 
and the processing in step S2504 is not restricted to the 
20 example in Fig. 88, so the processing in step S2504 may be 
executed first, or the processing in step S2503 and the 
processing in step S2504 may be executed simultaneously. 

Next, in step S2505, the normal equation generating 
unit 252 5 generates a normal equation table based on the 
2 5 input pixel value table generated by the input pixel value 
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acquiring unit 2523 at the processing in step S2503, and the 
integral component table generated by the integral component 
calculation unit 2524 at the processing in step S2504. 

Specifically, in this case, the features wi of the 
following Expression (96) corresponding to the above 
Expression (95) are calculated using the least square method. 
A normal equation corresponding to this is represented as 
the following Expression (97) . 
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. . . (97) 

If we define each matrix of the normal equation shown 
in Expression (97) as the following Expressions (98) through 
(100) , the normal equation is represented as the following 
Expression (101) . 
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As shown in Expression (99)', the respective components 
of the matrix Wmat are the features w± to be obtained. 
Accordingly, in Expression (101) , if the matrix Sma T of the 
5 left side and the matrix Pmat of the right side are 

determined, the matrix Wmat (i.e., features w±) may be 
calculated with the matrix solution. 

Specifically, as shown in Expression (98) , the 
respective components of the matrix Sma T may be calculated as 

10 long as the above integral components Si (1) are known. The 
integral components Si (1) are included in the integral 
component table supplied from the integral component 
calculation unit 2524, so the normal equation generating 
unit 2525 can calculate each component of the matrix Smat 

15 using the integral component table. 

Also, as shown in Expression (100) , the respective 
components of the matrix Pmat may be calculated as long as 
the integral components Si (1) and the input pixel values P 
(1) are known. The integral components Si (1) is the same 

2 0 as those included in the respective components of the matrix 
Smat/ also the input pixel values P (1) are included in the 
input pixel value table supplied from the input pixel value 
acquiring unit 2523, so the normal equation generating unit 
2525 can calculate each component of the matrix Pmat using 

25 the integral component table and input pixel value table. 
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Thus, the normal equation generating unit 2525 
calculates each component of the matrix Sma T and matrix Pmat/ 
and outputs the calculated results (each component of the 
matrix Smat and matrix Pmat) to the approximation function 
5 generating unit 2526 as a normal equation table. 

Upon the normal equation table being output from the 
normal equation generating unit 2526, in step S2506, the 
approximation function generating unit 2526 calculates the 
features wi (i.e., the coefficients wi of the approximation 
10 function f (x, y, t) ) serving as the respective components of 
the matrix Wma T in the above Expression (101) based on the 
normal equation table. 

Specifically, the normal equation in the above 
Expression (101) can be transformed as the following 
15 Expression (102) . 

W MAT = S MAT P MAT 

. . . (102) 

In Expression (102), the respective components of the 
matrix Wmat in the left side are the features wi to be 
20 obtained. The respective components regarding the matrix 

Smat and matrix Pmat are included in the normal equation table 
supplied from the normal equation generating unit 2525. 
Accordingly, the approximation function generating unit 2526 
calculates the matrix Wmat by calculating the matrix in the 
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right side of Expression (102) using the normal equation 
table, and outputs the calculated results (features Wj.) to 
the image generating unit 103. 

In step S2507, the approximation function generating 
5 unit 2 52 6 determines regarding whether or not the processing 
of all the pixels has been completed. 

In step S2507, in the event that determination is made 
that the processing of all the pixels has not been completed, 
the processing returns to step S2502, wherein the subsequent 
10 processing is repeatedly performed. That is to say, the 
pixels that have not become a pixel of interest are 
sequentially taken as a pixel of interest, and the 
processing in step S2502 through S2507 is repeatedly 
performed. 

15 In the event that the processing of all the pixels has 

been completed (in step S2507, in the event that 
determination is made that the processing of all the pixels 
has been completed) , the estimating processing of the actual 
world 1 ends . 

20 As described above, the three-dimensional approximating 

method takes three-dimensional integration effects in the 
temporal and spatial directions into consideration instead 
of one -dimensional or two-dimensional integration effects, 
and accordingly, can estimate the light signals in the 

25 actual world 1 more accurately than the one-dimensional 
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approximating method and two-dimensional polynomial 
approximating method . 

Next, description will be made regarding an embodiment 
of the image generating unit 103 (Fig. 3) with reference to 
5 Fig. 8 9 through Fig. 110. 

Fig. 89 is a diagram for describing the principle of 
the present embodiment . 

As shown in Fig. 89, the present embodiment is based on 
condition that the actual world estimating unit 102 employs 
10 a function approximating method. That is to say, let us say 
that the signals in the actual world 1 (distribution of 
light intensity) serving as an image cast in the sensor 2 
are represented with a predetermined function F, it is an 
assumption for the actual world estimating unit 102 to 
15 estimate the function F by approximating the function F with 
a predetermined function f using the input image (pixel 
value P) output from the sensor 2 and the data continuity 
information output from the data continuity detecting unit 
101 . 

20 Note that hereafter, with description of the present 

embodiment, the signals in the actual world 1 serving as an 
image are particularly referred to as light signals, and the 
function F is particularly referred to as a light signal 
function F. Also, the function f is particularly referred 

25 to as an approximation function f . 
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With the present embodiment, the image generating unit 
103 integrates the approximation function f with a 
predetermined time- space range using the data continuity 
information output from the data continuity detecting unit 
5 101, and the actual world estimating information (in the 
example in Fig. 89, the features of the approximation 
function f or approximation function f of which the features 
are identified) output from the actual world estimating unit 
102 based on such an assumption, and outputs the integral 
10 value as an output pixel value M (output image) . Note that 
with the present embodiment, an input pixel value is 
described as P, and an output pixel value is described as M 
in order to distinguish an input image pixel from an output 
image pixel . 

15 In other words, upon the light signal function F being 

integrated once, the light signal function F becomes an 
input pixel value P, the light signal function F is 
estimated from the input pixel value P (approximated with 
the approximation function f ) , the estimated light signal 

20 function F(i.e., approximation function f) is integrated 
again, and an output pixel value M is generated. 
Accordingly, hereafter, integration of the approximation 
function f executed by the image generating unit 103 is 
re ferred to as reintegration. Also, the present embodiment 

25 is referred to as a reintegration method. 
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Note that as described later, with the reintegration 
method, the integral range of the approximation function f 
in the event that the output pixel value M is generated is 
not restricted to the integral range of the light signal 
5 function F in the event that the input pixel value P is 

generated (i.e., the vertical width and horizontal width of 
the detecting element of the sensor 2 for the spatial 
direction, the exposure time of the sensor 2 for the 
temporal direction) , an arbitrary integral range may be 
10 employed. 

For example, in the event that the output pixel value M 
is generated, varying the integral range in the spatial 
direction of the integral range of the approximation 
function f enables the pixel pitch of an output image 

15 according to the integral range thereof to be varied. That 
is to say, creation of spatial resolution is available. 

In the same way, for example, in the event that the 
output pixel value M is generated, varying the integral 
range in the temporal direction of the integral range of the 

2 0 approximation function f enables creation of temporal 
resolution. 

Hereafter, description will be made individually 
regarding three specific methods of such a reintegration 
method with reference to the drawings. 
25 That is to say, three specific methods are 
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reintegration methods corresponding to three specific 
methods of the function approximating method (the above 
three specific examples of the embodiment of the actual 
world estimating unit 102) respectively. 
5 Specifically, the first method is a reintegration 

method corresponding to the above one-dimensional 
approximating method (one method of the function 
approximating method) . Accordingly, with the first method, 
one -dimensional reintegration is performed, so hereafter, 

10 such a reintegration method is referred to as a one- 
dimensional reintegration method. 

The second method is a reintegration method 
corresponding to the above two-dimensional polynomial 
approximating method (one method of the function 

15 approximating method) . Accordingly, with the second method, 
two-dimensional reintegration is performed, so hereafter, 
such a reintegration method is referred to as a two- 
dimensional reintegration method. 

The third method is a reintegration method 

2 0 corresponding to the above three-dimensional approximating 
method (one method of the function approximating method) . 
Accordingly, with the third method, three-dimensional 
reintegration is performed, so hereafter, such a 
reintegration method is referred to as a three-dimensional 

2 5 reintegration method. 
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Hereafter, description will be made regarding each 
details of the one -dimensional reintegration method, two- 
dimensional reintegration method, and three-dimensional 
reintegration method in this order. 
5 First, the one-dimensional reintegration method will be 

described. 

With the one -dimensional reintegration method, it is 
assumed that the approximation function f (x) is generated 
using the one-dimensional approximating method. 

10 That is to say, it is an assumption that a one- 

dimensional waveform (with description of the reintegration 
method, a waveform projected in the X direction of such a 
waveform is referred to as an X cross-sectional waveform 
F (x) ) wherein the light signal function F (x, y, t) of which 

15 variables are positions x, y, and z on the three-dimensional 
space, and a point-in-time t is projected in a predetermined 
direction (for example, X direction) of the X direction, Y 
direction, and Z direction serving as the spatial direction, 
and t direction serving as the temporal direction, is 

2 0 approximated with the approximation function f (x) serving as 
an n-dimensional (n is an arbitrary integer) polynomial. 

In this case, with the one-dimensional reintegration 
method, the output pixel value M is calculated such as the 
following Expression (103). 
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M = G e x f Xe f (x)dx 

. . . (103) 

Note that in Expression (103) , x s represents an 
integration start position, x e represents an integration end 
5 position. Also, G e represents a predetermined gain. 

Specifically, for example, let us say that the actual 
world estimating unit 102 has already generated the 
approximation function f (x) (the approximation function f (x) 
of the X cross-sectional waveform F (x) ) such as shown in Fig. 
10 90 with a pixel 3101 (pixel 3101 corresponding to a 

predetermined detecting element of the sensor 2) such as 
shown in Fig. 90 as a pixel of interest. 

Note that with the example in Fig. 90, the pixel value 
(input pixel value) of the pixel 3101 is taken as P, and the 
15 shape of the pixel 3101 is taken as a square of which one 
side is 1 in length. Also, of the spatial directions, the 
direction in parallel with one side of the pixel 3101 
(horizontal direction in the drawing) is taken as the X 
direction, and the direction orthogonal to the X direction 
20 (vertical direction in the drawing) is taken as the Y 
direction. 

Also, on the lower side in Fig. 90, the coordinates 
system (hereafter, referred to as a pixel -of -interest 
coordinates system) in the spatial directions (X direction 
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and Y direction) of which the center of the pixel 3101 is 
taken as the origin, and the pixel 3101 in .the coordinates 
system are shown. 

Further, on the upward direction in Fig. 90, a graph 
5 representing the approximation function f (x) at y = 0 (y is 
a coordinate value in the Y direction in the pixel-of- 
interest coordinates system shown on the lower side in the 
drawing) is shown. In this graph, the axis in parallel with 
the horizontal direction in the drawing is the same axis as 

10 the x axis in the X direction in the pixel-of -interest 

coordinates system shown on the lower side in the drawing 
(the origin is also the same) , and also the axis in parallel 
with the vertical direction in the drawing is taken as an 
axis representing pixel values. 

15 In this case, the relation of the following Expression 

(104) holds between the approximation function f (x) and the 
pixel value P of the pixel 3101. 



3101 has continuity of data in the spatial direction 
represented with the gradient G f . Further, let us say that 
the data continuity detecting unit 101 (Fig. 89) has already 
output the angle 0 such as shown in Fig. 90 as data 




. . . (104) 



20 



Also, as shown in Fig. 90, let us say that the pixel 
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continuity information corresponding to continuity of data 
represented with the gradient Gf . 

In this case, for example, with the one -dimensional 
reintegration method, as shown in Fig. 91, four pixels 3111 
5 through 3114 can be newly created in a range of -0.5 through 
0.5 in the X direction, and also in a range of -0.5 through 
0.5 in the Y direction (in the range where the pixel 3101 in 
Fig. 90 is positioned) . 

Note that on the lower side in Fig. 91, the same pixel - 

10 of -interest coordinates system as that in Fig. 90, and the 
pixels 3111 through 3114 in the pixel -of -interest 
coordinates system thereof are shown. Also, on the upper 
side in Fig. 91, the same graph (graph representing the 
approximation function f (x) at y = 0) as that in Fig. 90 is 

15 shown. 

Specifically, as shown in Fig. 91, with the one- 
dimensional reintegration method, calculation of the pixel 
value M (1) of the pixel 3111 using the following Expression 
(105) , calculation of the pixel value M (2) of the pixel 
20 3112 using the following Expression (106) , calculation of 

the pixel value M (3) of the pixel 3113 using the following 
Expression (107) , and calculation of the pixel value M (4) 
of the pixel 3114 using the following Expression (108) are 
available respectively . 
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M(1) =2x f Xe1 f (x)dx 



. . . (105) 



J. X 6 2 
f (x)dx 
x s2 



. . . (106) 



M(3) =2x f Xe3 f ( X )dx 



. (107) 



f (x)dx 



x s4 

. . . (108) 

Note that x si in Expression (105) , x s2 in Expression 
10 (106) , x s3 in Expression (107) , and x s4 in Expression (108) 
each represent the integration start position of the 
corresponding expression. Also, x e i in Expression (105) , x e2 
in Expression (106) , x e 3 in Expression (107) , and x e 4 in 
Expression (108) each represent the integration end position 
15 of the corresponding expression. 

The integral range in the right side of each of 
Expression (105) through Expression (108) becomes the pixel 
width (length in the X direction) of each of the pixel 3111 
through pixel 3114. That is to say, each of x e i - x s i, x e2 - 
2 0 x s2 , x e 3 - x s3 , and x e4 - x s4 becomes 0.5. 

However, in this case, it can be conceived that a one- 
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dimensional waveform having the same form as that in the 
approximation function f (x) at y = 0 continues not in the Y 
direction but in the direction of data continuity 
represented with the gradient G f (i.e., angle 0 direction) 
5 (in fact, a waveform having the same form as the X cross- 
sectional waveform F (x) at y = 0 continues in the direction 
of continuity) . That is to say, in the case in which a 
pixel value f (0) in the origin (0, 0) in the pixel-of- 
interest coordinates system in Fig. 91 (center of the pixel 

10 3101 in Fig. 90) is taken as a pixel value fl, the direction 
where the pixel value fl continues is not the Y direction 
but the direction of data continuity represented with the 
gradient G f (angle 6 direction) . 

In other words, in the case of conceiving the waveform 

15 of the approximation function f (x) in a predetermined 

position y in the Y direction (however, y is a numeric value 
other than zero) , the position corresponding to the pixel 
value fl is not a position (0, y) but a position (C x (y) , y) 
obtained by moving in the X direction from the position (0, 

2 0 y) by a predetermined amount (here, let us say that such an 
amount is also referred to as a shift amount. Also, a shift 
amount is an amount depending on the position y in the Y 
direction, so let us say that this shift amount is described 
as C x (y) ) . 

25 Accordingly, as the integral range of the right side of 
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each of the above Expression (105) through Expression (108) , 
the integral range needs to be set in light of the position 
y in the Y direction where the center of the pixel value M 
(1) to be obtained (however, 1 is any integer value of 1 
5 through 4) exists, i.e., the shift amount C x (y) . 

Specifically, for example, the position y in the Y 
direction where the centers of the pixel 3111 and pixel 3112 
exist is not y = 0 but y = 0.25. 

Accordingly, the waveform of the approximation function 

10 f (x) at y = 0.25 is equivalent to a waveform obtained by 

moving the waveform of the approximation function f (x) at y 
= 0 by the shift amount C x (0.25) in the X direction. 

In other words, in the above Expression (105) , if we 
say that the pixel value M (1) as to the pixel 3111 is 

15 obtained by integrating the approximation function f (x) at y 
= 0 with a predetermined integral range (from the start 
position x s i to the end position x e i) , the integral range 
thereof becomes not a range from the start position x si = - 
0.5 to the end position x e i = 0 (a range itself where the 

20 pixel 3111 occupies in the X direction) but the range shown 
in Fig. 91, i.e., from the start position x si = -0.5 + C x 
(0.25) to the end position x e i = 0 + C x (0.25) (a range where 
the pixel 3111 occupies in the X direction in the event that 
the pixel 3111 is tentatively moved by the shift amount C x 

25 (0.25) ) . 
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Similarly, in the above Expression (106) , if we say 
that the pixel value M (2) as to the pixel 3112 is obtained 
by integrating the approximation function f (x) at y = 0 with 
a predetermined integral range (from the start position x S 2 
5 to the end position x e2 ) , the integral range thereof becomes 
not a range from the start position x s2 = 0 to the end 
position x e2 =0.5 (a range itself where the pixel 3112 
occupies in the X direction) but the range shown in Fig. 91, 
i.e., from the start position x s2 = 0 + C x (0.25) to the end 

10 position x e i = 0 . 5 + C x (0.25) (a range where the pixel 3112 
occupies in the X direction in the event that the pixel 3112 
is tentatively moved by the shift amount C x (0.25)) . 

Also, for example, the position y in the Y direction 
where the centers of the pixel 3113 and pixel 3114 exist is 

15 not y = 0 but y = -0.25. 

Accordingly, the waveform of the approximation function 
f (x) at y = -0.25 is equivalent to a waveform obtained by 
moving the waveform of the approximation function f (x) at y 
= 0 by the shift amount C x (-0.25) in the X direction. 

2 0 In other words, in the above Expression (107) , if we 

say that the pixel value M (3) as to the pixel 3113 is 
obtained by integrating the approximation function f (x) at y 
= 0 with a predetermined integral range (from the start 
position x s3 to the end position x e 3) / the integral range 

25 thereof becomes not a range from the start position x s3 = - 
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0.5 to the end position x e3 = 0 (a range itself where the 
pixel 3113 occupies in the X direction) but the range shown 
in Fig. 91, i.e., from the start position x s3 = -0.5 + C x (- 

0. 25) to the end position x e3 = 0 + C x (-0.25) (a range where 
5 the pixel 3113 occupies in the X direction in the event that 

the pixel 3113 is tentatively moved by the shift amount C x 
(-0.25) ) . 

Similarly, in the above Expression (108) , if we say 
that the pixel value M (4) as to the pixel 3114 is obtained 

10 by integrating the approximation function f (x) at y = 0 with 
a predetermined integral range (from the start position x s4 
to the end position x e4 ) / the integral range thereof becomes 
not a range from the start position x s4 = 0 to the end 
position x e4 =0.5 (a range itself where the pixel 3114 

15 occupies in the X direction) but the range shown in Fig. 91, 

1. e., from the start position x s4 = 0 + C x (-0.25) to the end 
position x e i = 0.5 + C x (-0.25) (a range where the pixel 3114 
occupies in the X direction in the event that the pixel 3114 
is tentatively moved by the shift amount C x (-0.25)) . 

20 Accordingly, the image generating unit 102 (Fig. 89) 

calculates the above Expression (105) through Expression 
(108) by substituting the corresponding integral range of 
the above integral ranges for each of these expressions, and 
outputs the calculated results of these as the output pixel 

2 5 values M (1) through M (4) . 
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Thus, the image generating unit 102 can create four 
pixels having higher spatial resolution than that of the 
output pixel 3101, i.e., the pixel 3111 through pixel 3114 
(Fig. 91) by employing the one -dimensional reintegration 
5 method as a pixel at the output pixel 3101 (Fig. 90) from 
the sensor 2 (Fig. 89) . Further, though not shown in the 
drawing, as described above, the image generating unit 102 
can create a pixel having an arbitrary powered spatial 
resolution as to the output pixel 3101 without deterioration 

10 by appropriately changing an integral range, in addition to 
the pixel 3111 through pixel 3114. 

Fig. 92 represents a configuration example of the image 
generating unit 103 employing such a one-dimensional 
reintegration method. 

15 As shown in Fig. 92, the image generating unit 103 

shown in this example includes a conditions setting unit 
3121, features storage unit 3122, integral component 
calculation unit 3123, and output pixel value calculation 
unit 3124. 

20 The conditions setting unit 3121 sets the number of 

dimensions n of the approximation function f (x) based on the 
actual world estimating information (the features of the 
approximation function f (x) in the example in Fig. 92) 
supplied from the actual world estimating unit 102. 

25 The conditions setting unit 3121 also sets an integral 
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range in the case of reintegrating the approximation 
function f (x) (in the case of calculating an output pixel 
value) . Note that an integral range set by the conditions 
setting unit 3121 does not need to be the width of a pixel. 
5 For example, the approximation function f (x) is integrated 
in the spatial direction (X direction) , and accordingly, a 
specific integral range can be determined as long as the 
relative size (power of spatial resolution) of an output 
pixel (pixel to be calculated by the image generating unit 

10 103) as to the spatial size of each pixel of an input image 
from the sensor 2 (Fig. 89) is known. Accordingly, the 
conditions setting unit 3121 can set, for example, a spatial 
resolution power as an integral range. 

The features storage unit 3122 temporally stores the 

15 features of the approximation function f (x) sequentially 
supplied from the actual world estimating unit 102. 
Subsequently, upon the features storage unit 3122 storing 
all of the features of the approximation function f (x) , the 
features storage unit 3122 generates a features table 

20 including all of the features of the approximation function 
f (x) , and supplies this to the output pixel value 
calculation unit 3124. 

Incidentally, as described above, the image generating 
unit 103 calculates the output pixel value M using the above 

25 Expression (103), but the approximation function f (x) 
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included in the right side of the above Expression (103) is 
represented as the following Expression (109) specifically, 
n 

f (x) = I Wj xx'dx 

i=0 

. . . (109) 

5 Note that in Expression (109) , w ± represents the 

features of the approximation function f (x) supplied from 
the actual world estimating unit 102. 

Accordingly, upon the approximation function f (x) of 
Expression (109) being substituted for the approximation 
10 function f (x) of the right side of the above Expression 
(103) so as to expand (calculate) the right side of 
Expression (103) , the output pixel value M is represented as 
the following Expression (110) . 

i+1 i+1 
n X — X 

M = G e x| o W i x i 

n 

= I Wj xkj (x s , x e ) 

i=0 

15 ... (HO) 

In Expression (110) , Ki (x s> x e ) represent the integral 
components of the i -dimensional term. That is to say, the 
integral components Ki (x a , x e ) are such as shown in the 
following Expression (111) . 

i+1 



k;(x s , x e ) =G e x ' <e Xs 



20 



i+1 
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The integral component calculation unit 3123 calculates 
the integral components K ± (x S/ x e ) . 

Specifically, as shown in Expression (111) , the 
5 integral components Ki (x S/ x e ) can be calculated as long as 
the start position x s and end position x e of an integral 
range, gain G e , and i of the i -dimensional term are known. 

Of these, the gain G e is determined with the spatial 
resolution power (integral range) set by the conditions 
10 setting unit 3121. 

The range of i is determined with the number of 
dimensions n set by the conditions setting unit 3121. 

Also, each of the start position x s and end position x e 
of an integral range is determined with the center pixel 
15 position (x, y) and pixel width of an output pixel to be 

generated from now, and the shift amount C x (y) representing 
the direction of data continuity. Note that (x, y) 
represents the relative position from the center position of 
a pixel of interest when the actual world estimating unit 
20 102 generates the approximation function f (x) . 

Further, each of the center pixel position (x, y) and 
pixel width of an output pixel to be generated from now is 
determined with the spatial resolution power (integral 
range) set by the conditions setting unit 3121. 
25 Also, with the shift amount C x (y) , and the angle 9 
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supplied from the data continuity detecting unit 101, the 
relation such as the following Expression (112) and 
Expression (113) holds, and accordingly, the shift amount C x 
(y) is determined with the angle 0. 

dy 
dx 

. . . (112) 



G f = tan 6 = 



. . . (113) 

Note that in Expression (112) , G f represents a gradient 
10 representing the direction of data continuity, 0 represents 
an angle (angle generated between the X direction serving as 
one direction of the spatial directions and the direction of 
data continuity represented with a gradient G f ) of one of 
the data continuity information output from the data 
15 continuity detecting unit 101 (Fig. 89) . Also, dx 

represents the amount of fine movement in the X direction, 
and dy represents the amount of fine movement in the Y 
direction (spatial direction perpendicular to the X 
direction) as to the dx. 
2 0 Accordingly, the integral component calculation unit 

3123 calculates the integral components Ki (x s , x e ) based on 
the number of dimensions and spatial resolution power 
(integral range) set by the conditions setting unit 3121, 
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and the angle 0 of the data continuity information output 
from the data continuity detecting unit 101, and supplies 
the calculated results to the output pixel value calculation 
unit 3124 as an integral component table. 
5 The output pixel value calculation unit 3124 calculates 

the right side of the above Expression (110) using the 
features table supplied from the features storage unit 3122 
and the integral component table supplied from the integral 
component calculation unit 3123, and outputs the calculation 

10 result as an output pixel value M. 

Next, description will be made regarding image 
generating processing (processing in step S103 in Fig. 29) 
by the image generating unit 103 (Fig. 92) employing the 
one -dimensional reintegration method with reference to the 

15 flowchart in Fig. 93. 

For example, now, let us say that the actual world 
estimating unit 102 has already generated the approximation 
function f (x) such as shown in Fig. 90 while taking the 
pixel 3101 such as shown in Fig. 90 described above as a 

20 pixel of interest at the processing in step S102 in Fig. 29 
described above. 

Also, let us say that the data continuity detecting 
unit 101 has already output the angle 0 such as shown in Fig. 
90 as data continuity information at the processing in step 

25 S101 in Fig. 29 described above. 
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In this case, the conditions setting unit 3121 sets 
conditions (the number of dimensions and an integral range) 
at step S3101 in Fig. 93. 

For example, now, let us say that 5 has been set as the 
5 number of dimensions, and also a spatial quadruple density 
(spatial resolution power to cause the pitch width of a 
pixel to become half power in the upper/lower/left/right 
sides) has been set as an integral range. 

That is to say, in this case, consequently, it has been 
10 set that the four pixel 3111 through pixel 3114 are created 
newly in a range of -0.5 through 0.5 in the X direction, and 
also a range of -0.5 through 0.5 in the Y direction (in the 
range of the pixel 3101 in Fig. 90) , such as shown in Fig. 
91. 

15 In step S3102, the features storage unit 3122 acquires 

the features of the approximation function f (x) supplied 
from the actual world estimating unit 102, and generates a 
features table. In this case, coefficients w 0 through w 5 of 
the approximation function f (x) serving as a five- 

2 0 dimensional polynomial are supplied from the actual world 

estimating unit 102, and accordingly, (w 0 , w i# w 2 , w 3 , w 4 , w 5 ) 
is generated as a features table. 

In step S3103, the integral component calculation unit 
3123 calculates integral components based on the conditions 

25 (the number of dimensions and integral range) set by the 
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conditions setting unit 3121, and the data continuity 
information (angle 6) supplied from the data continuity 
detecting unit 101, and generates an integral component 
table . 

Specifically, for example, if we say that the 
respective pixels 3111 through 3114, which are to be 
generated from now, are appended with numbers (hereafter, 
such a number is referred to as a mode number) 1 through 4, 
the integral component calculation unit 3123 calculates the 
integral components Ki (x s , x e ) of the above Expression (111) 
as a function of 1 (however, 1 represents a mode number) 
such as integral components Ki (1) shown in the left side of 
the following Expression (114) . 
Ki (1) = Ki (x s , x e ) 

. . . (114) 

Specifically, in this case, the integral components Ki 
(1) shown in the following Expression (115) are calculated. 

kjd) = kj (-0. 5-C x (-0. 25) , 0-C x (-0. 25) ) 
k ; (2) = kj (0-C x (-0. 25) , 0. 5-C x (-0. 25) ) 
kj (3) = kj (-0. 5-C x (0. 25) , 0-C x (0. 25) ) 
kj (4) = kj (0-C x (0. 25) , 0. 5-C x (0. 25) ) 

. . . (115) 

Note that in Expression (115) , the left side represents 
the integral components K± (1) , and the right side 
represents the integral components Ki (x s , x e ) . That is to 
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say, in this case, 1 is any one of 1 through 4, and also i 
is any one of 0 through 5, and accordingly, 24 Ki (1) in 
total of 6 Ki (1), 6 Ki (2), 6 Ki (3), and 6 Ki (4) are 
calculated . 

5 More specifically, first, the integral component 

calculation unit 3123 calculates each of the shift amounts 
C x (-0.25) and C x (0.25) from the above Expression (112) and 
Expression (113) using the angle 0 supplied from the data 
continuity detecting unit 101. 

10 Next, the integral component calculation unit 3123 

calculates the integral components Ki (x s , x e ) of each right 
side of the four expressions in Expression (115) regarding i 
= 0 through 5 using the calculated shift amounts C x (-0.25) 
and C x (0.25). Note that with this calculation of the 

15 integral components Ki (x s , x e ) , the above Expression (111) 
is employed. 

Subsequently, the integral component calculation unit 
3123 converts each of the 24 integral components Ki (x s , x e ) 
calculated into the corresponding integral components Ki (1) 
20 in accordance with Expression (115) , and generates an 
integral component table including the 24 integral 
components Ki (1) converted (i.e., 6 K± (1), 6 Ki (2), 6 Ki 
(3) , and 6 K± (4) ) . 

Note that the sequence of the processing in step S3 102 
25 and the processing in step S3103 is not restricted to the 
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example in Fig. 93, the processing in step S3 103 may be 
executed first, or the processing in step S3 102 and the 
processing in step S3103 may be executed simultaneously. 

Next, in step S3 104, the output pixel value calculation 
5 unit 3124 calculates the output pixel values M (1) through M 
(4) respectively based on the features table generated by 
the features storage unit 3122 at the processing in step 
S3102, and the integral component table generated by the 
integral component calculation unit 3123 at the processing 

10 in step S3103 . 

Specifically, in this case, the output pixel value 
calculation unit 3124 calculates each of the pixel value M 
(1) of the pixel 3111 (pixel of mode number 1) , the pixel 
value M (2) of the pixel 3112 (pixel of mode number 2) , the 

15 pixel value M (3) of the pixel 3113 (pixel of mode number 3), 
and the pixel value M (4) of the pixel 3114 (pixel of mode 
number 4) by calculating the right sides of the following 
Expression (116) through Expression (119) corresponding to 
the above Expression (110) . 



M(1) 



I Wjkjd) 



20 



i=0 



. . . (116) 



M(2) 



I Wjkj(2) 



i=0 



. . . (117) 
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5 



M(3) 



I WjkjO) 



i=0 



. . . (118) 



5 



M(4) 



i=0 



I Wjki (4) 



. . . (119) 



In step S3105, the output pixel value calculation unit 
3124 determines regarding whether or not the processing of 
all the pixels has been completed. 

In step S3 105, in the event that determination is made 
that the processing of all the pixels has not been completed, 
the processing returns to step S3102, wherein the subsequent 
processing is repeatedly performed. That is to say, the 
pixels that have not become a pixel of interest are 
sequentially taken as a pixel of interest, and the 
processing in step S3102 through S3104 is repeatedly 
performed . 

In the event that the processing of all the pixels has 
been completed (in step S3 105, in the event that 
determination is made that the processing of all the pixels 
has been completed) , the output pixel value calculation unit 
3124 outputs the image in step S3106. Then, the image 
generating processing ends. 

Next, description will be made regarding the 
differences between the output image obtained by employing 
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the one-dimensional reintegration method and the output 
image obtained by employing another method (conventional 
classification adaptation processing) regarding a 
predetermined input image with reference to Fig. 94 through 
5 Fig. 101. 

Fig. 94 is a diagram illustrating the original image of 
the input image, and Fig. 95 illustrates image data 
corresponding to the original image in Fig. 94. In Fig. 95, 
the axis in the vertical direction in the drawing represents 

10 pixel values, and the axis in the lower right direction in 
the drawing represents the X direction serving as one 
direction of the spatial directions of the image, and the 
axis in the upper right direction in the drawing represents 
the Y direction serving as the other direction of the 

15 spatial directions of the image. Note that the respective 
axes in later-described Fig. 97, Fig. 99, and Fig. 101 
corresponds to the axes in Fig. 95. 

Fig. 96 is a diagram illustrating an example of an 
input image. The input image illustrated in Fig. 96 is an 

2 0 image generated by taking the mean of the pixel values of 

the pixels belonged to a block made up of 2 x 2 pixels shown 
in Fig. 94 as the pixel value of one pixel. That is to say, 
the input image is an image obtained by integrating the 
image shown in Fig. 94 in the spatial direction, which 

25 imitates the integration property of a sensor. Also, Fig. 
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97 illustrates image data corresponding to the input image 
in Fig. 96. 

The original image illustrated in Fig. 94 includes a 
fine-line image inclined almost 5° clockwise from the 
5 vertical direction. Similarly, the input image illustrated 
in Fig. 96 includes a fine-line image inclined almost 5° 
clockwise from the vertical direction. 

Fig. 98 is a diagram illustrating an image (hereafter, 
the image illustrated in Fig. 98 is referred to as the 
10 original image) obtained by subjecting the input image 

illustrated in Fig. 96 to conventional class classification 
adaptation processing. Also, Fig. 99 illustrates image data 
corresponding to the conventional image. 

Note that the class classification adaptation 
15 processing is made up of classification processing and 
adaptation processing, data is classified based on the 
property thereof by the class classification processing, and 
is subjected to the adaptation processing for each class, as 
described above. With the adaptation processing, for 
20 example, a low-quality or standard-quality image is 

subjected to mapping using a predetermined tap coefficient 
so as to be converted into a high-quality image. 

Fig. 100 is a diagram illustrating an image (hereafter, 
the image illustrated in Fig. 100 is referred to as a 
25 reintegration image) obtained by applying the one- 
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dimensional reintegration method, to the input image 
illustrated in Fig. 96. Also, Fig. 101 illustrates image 
data corresponding to the reintegration image. 

It can be understood that upon the conventional image 
5 in Fig. 98 being compared with the reintegration image in 
Fig. 100, a fine-line image is different from that in the 
original image in Fig. 94 in the conventional image, but on 
the other hand, the fine -line image is almost the same as 
that in the original image in Fig. 94 in the reintegration 
10 image. 

This difference is caused by a difference wherein the 
conventional class classification adaptation processing is a 
method for performing processing on the basis (origin) of 
the input image in Fig. 96, but on the other hand, the one- 

15 dimensional reintegration method is a method for estimating 
the original image in Fig. 94 (generating the approximation 
function f (x) corresponding to the original image) in light 
of continuity of a fine line, and performing processing 
(performing reintegration so as to calculate pixel values) 

20 on the basis (origin) of the original image estimated. 

Thus, with the one-dimensional reintegration method, 
the output image (pixel value) is generated by integrating 
the approximation function f (x) in an arbitrary range, using 
an approximation function f (x) such as a primary polynomial 

25 generated by a one-dimensional approximation method (the 
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approximation function f (x) of the X-cross section waveform 
F (x) of the actual world) as a standard (origin). 

Accordingly, with the one-dimensional reintegration 
method, it becomes possible to output an image more similar 
5 to the original image (the light signal in the actual world 
1 which is to be cast in the sensor 2) in comparison with 
the conventional other methods. 

Also, with the one-dimensional reintegration method, as 
described above, the integral range is arbitrary, and 
10 accordingly, it becomes possible to create resolution 

(temporal resolution or spatial resolution) different from 
the resolution of an input image by varying the integral 
range. That is to say, it becomes possible to generate an 
image having arbitrary powered resolution as well as an 
15 integer value as to the resolution of the input image. 

Further, the one-dimensional reintegration method 
enables calculation of an output image (pixel values) with 
less calculation processing amount than other reintegration 
methods . 

20 Next, description will be made regarding a two- 

dimensional reintegration method with reference to Fig. 102 
through Fig. 108. 

The two-dimensional reintegration method is based on 
condition that the approximation function f (x, y) has been 

2 5 generated with the two-dimensional polynomial approximating 
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method. 

That is to say, for example, it is an assumption that 
the image function F(x, y, t) representing the light signal 
in the actual world 1 (Fig. 89) having continuity in the 
5 spatial direction represented with the gradient G F has been 
approximated with a waveform projected in the spatial 
directions (X direction and Y direction), i.e., the waveform 
F(x, y) on the X-Y plane has been approximated with the 
approximation function f (x, y) serving as a n-dimensional (n 
10 is an arbitrary integer) polynomial, such as shown in Fig. 
102 . 

In Fig. 102, the horizontal direction represents the X 
direction serving as one direction in the spatial directions, 
the upper right direction represents the Y direction serving 

15 as the other direction in the spatial directions, and the 

vertical direction represents light levels, respectively in 
the drawing. G F represents gradient as continuity in the 
spatial directions . 

Note that with the example in Fig. 102, the direction 

20 of continuity is taken as the spatial directions (X 

direction and Y direction) , so the projection function of a 
light signal to be approximated is taken as the function F(x, 
y) , but as described later, the function F(x, t) or function 
F(y, t) may be a target of approximation according to the 

25 direction of continuity. 
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In the case of the example in Fig. 102, with the two- 
dimensional reintegration method, the output pixel value M 
is calculated as the following Expression (120) . 

M = G e x f ye f Xe f (x,y)dxdy 

. . . (120) 

Note that in Expression (120) , y s represents an 
integration start position in the Y direction, and y e 
represents an integration end position in the Y direction. 
Similarly, x s represents an integration start position in 
the X direction, and x e represents an integration end 
position in the X direction. Also, G e represents a 
predetermined gain. 

In Expression (120) , an integral range can be set 
arbitrarily, and accordingly, with the two-dimensional 
reintegration method, it becomes possible to create pixels 
having an arbitrary powered spatial resolution as to the 
original pixels (the pixels of an input image from the 
sensor 2 (Fig. 89)) without deterioration by appropriately 
changing this integral range. 

Fig. 103 represents a configuration example of the 
image generating unit 103 employing the two-dimensional 
reintegration method. 

As shown in Fig. 103, the image generating unit 103 in 
this example includes a conditions setting unit 3201, 
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features storage unit 32 02, integral component calculation 
unit 3203, and output pixel value calculation unit 3204. 

The conditions setting unit 3201 sets the number of 
dimensions n of the approximation function f (x, y) based on 
5 the actual world estimating information (with the example in 
Fig. 103, the features of the approximation function f (x, 
y) ) supplied from the actual world estimating unit 102. 

The conditions setting unit 3201 also sets an integral 
range in the case of reintegrating the approximation 

10 function f (x, y) (in the case of calculating an output pixel 
value) . Note that an integral range set by the conditions 
setting unit 3201 does not need to be the vertical width or 
the horizontal width of a pixel. For example, the 
approximation function f (x, y) is integrated in the spatial 

15 directions (X direction and Y direction) , and accordingly, a 
specific integral range can be determined as long as the 
relative size (power of spatial resolution) of an output 
pixel (pixel to be generated from now by the image 
generating unit 103) as to the spatial size of each pixel of 

20 an input image from the sensor 2 is known. Accordingly, the 
conditions setting unit 3201 can set, for example, a spatial 
resolution power as an integral range . 

The features storage unit 32 02 temporally stores the 
features of the approximation function f (x, y) sequentially 

25 supplied from the actual world estimating unit 102. 



- 302 - 

S04P0850 

Subsequently, upon the features storage unit 3202 storing 
all of the features of the approximation function f (x, y) , 
the features storage unit 32 02 generates a features table 
including all of the features of the approximation function 
5 f(x, y) , and supplies this to the output pixel value 
calculation unit 3204. 

Now, description will be made regarding the details of 
the approximation function f (x, y) . 

For example, now, let us say that the light signals 

10 (light signals represented with the wave F (x, y) ) in the 
actual world 1 (Fig. 89) having continuity in the spatial 
directions represented with the gradient G F shown in Fig. 
102 described above have been detected by the sensor 2 (Fig. 
89) , and have been output as an input image (pixel values) . 

15 Further, for example, let us say that the data 

continuity detecting unit 101 (Fig. 3) has subjected a 
region 3221 of an input image made up of 20 pixels in total 
(20 squares represented with a dashed line in the drawing) 
of 4 pixels in the X direction and also 5 pixels in the Y 

2 0 direction of this input image to the processing thereof, and 
has output an angle 6 (angle G generated between the 
direction of data continuity represented with the gradient 
G f corresponding to the gradient G F and the X direction) as 
one of data continuity information, as shown in Fig. 104. 

25 Note that as viewed from the actual world estimating 
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unit 102, the data continuity detecting unit 101 should 
simply output the angle 9 at a pixel of interest, and 
accordingly, the processing region of the data continuity 
detecting unit 101 is not restricted to the above region 
5 3 221 in the input image. 

Also, with the region 3221 in the input image, the 
horizontal direction in the drawing represents the X 
direction serving as one direction of the spatial directions, 
and the vertical direction in the drawing represents the Y 
10 direction serving the other direction of the spatial 
directions . 

Further, in Fig. 104, a pixel, which is the second 
pixel from the left, and also the third pixel from the 
bottom, is taken as a pixel of interest, and an (x, y) 
15 coordinates system is set so as to take the center of the 

pixel of interest as the origin (0, 0) . A relative distance 
(hereafter, referred to as a cross-sectional direction 
distance) in the X direction as to a straight line (straight 
line of the gradient G f representing the direction of data 
2 0 continuity) having an angle 0 passing through the origin (0, 
0 ) is taken as x ' . 

Further, in Fig. 104, the graph on the right side 
represents the approximation function f(x') serving as a n- 
dimensional (n is an arbitrary integer) polynomial, which is 
25 a function approximating a one-dimensional waveform 
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(hereafter, referred to as an X cross-sectional waveform 
F(x')) wherein the image function F (x, y, t) of which 
variables are positions x, y, and z on the three-dimensional 
space, and point-in- time t is projected in the X direction 
5 at an arbitrary position y in the Y direction. Of the axes 
in the graph on the right side, the axis in the horizontal 
direction in the drawing represents a cross-sectional 
direction distance, and the axis in the vertical direction 
in the drawing represents pixel values. 
10 In this case, the approximation function f(x') shown in 

Fig. 104 is a n-dimensional polynomial, so is represented as 
the following Expression (121) . 

n 

f(x') = W 0 +W 1 X , +W 2 x' + ---+W n X ,n = Z WjX" 

i=0 

. . . (121) 

15 Also, since the angle 6 is determined, the straight line 

having angle 0 passing through the origin (0, 0) is uniquely 
determined, and a position Xi in the X direction of the 
straight line at an arbitrary position y in the Y direction 
is represented as the following Expression (122) . However, 

20 in Expression (122), s represents cot 6. 
Xx = s x y 

. . . (122) 

That is to say, as shown in Fig. 104, a point on the 
straight line corresponding to continuity of data 
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represented with the gradient G f is represented with a 
coordinate value (xi, y) . 

The cross-sectional direction distance x' is 
represented as the following Expression (123) using 
5 Expression (122) . 

x 1 = x - Xi = x - s x y 

. . . (123) 

Accordingly, the approximation function f (x, y) at an 
arbitrary position (x, y) within the input image region 3221 
10 is represented as the following Expression (124) using 
Expression (121) and Expression (123) . 
n 

f (x B y) = Z Wj (x-sxy) 

i=0 

. . . (124) 

Note that in Expression (124) , w± represents the 
15 features of the approximation function f (x, y) . 

Now, description will return to Fig. 103, wherein the 
features Wi included in Expression (124) are supplied from 
the actual world estimating unit 102, and stored in the 
features storage unit 3202. Upon the features storage unit 
20 32 02 storing all of the features wi represented with 

Expression (124) , the features storage unit 3202 generates a 
features table including all of the features w ±/ and 
supplies this to the output pixel value calculation unit 
3204 . 
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Also, upon the right side of the above Expression (120) 
being expanded (calculated) by substituting the 
approximation function f (x, y) of Expression (124) for the 
approximation function f (x, y) in the right side of 
5 Expression (12 0) , the output pixel value M is represented as 
the following Expression (125) . 

u -A „ l(x,-sxy,) it2 -(x,- S xy s ) |t2 -(x rS xy e ) it? + (x s -sxy s ) i+; ) 
M = G e x I Wi x s(i+l)Ti+2) 

n 

= 1 Wj xkj (x s , x e , y Sf y e ) 

i=0 

. . . (125) 

In Expression (125) , K± (x s/ x e/ y s , y e ) represent the 
10 integral components of the i-dimensional term. That is to 
say, the integral components Ki (x s , x e/ y S / Ye) are such as 
shown in the following Expression (12 6) . 

k j (x s , x e , y s , y e ) 

_ p {(x e -SXy e ) i+2 ^^ 

s(i+1)(i+2) 

. . . (126) 

15 The integral component calculation unit 3203 calculates 

the integral components Ki (x S/ x e/ y S/ y e ) • 

Specifically, as shown in Expression (12 5) and 
Expression (12 6) , the integral components Ki (x s/ x e/ y S / Ye) 
can be calculated as long as the start position x s in the X 

20 direction and end position x e in the X direction of an 
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integral range, the start position y s in the Y direction and 
end position y e in the Y direction of an integral range, 
variable s, gain G e , and i of the i -dimensional term are 
known . 

5 Of these, the gain G e is determined with the spatial 

resolution power (integral range) set by the conditions 
setting unit 3201. 

The range of i is determined with the number of 
dimensions n set by the conditions setting unit 3201. 

10 A variable s is, as described above, cot 0, so is* 

determined with the angle 0 output from the data continuity 
detecting unit 101. 

Also, each of the start position x s in the X direction 
and end position x e in the X direction of an integral range, 

15 and the start position y s in the Y direction and end 

position y e in the Y direction of an integral range is 
determined with the center pixel position (x, y) and pixel 
width of an output pixel to be generated from now. Note 
that (x, y) represents a relative position from the center 

2 0 position of the pixel of interest when the actual world 
estimating unit 102 generates the approximation function 
f (x) . 

Further, each of the center pixel position (x, y) and 
pixel width of an output pixel to be generated from now is 
25 determined with the spatial resolution power (integral 
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range) set by the conditions setting unit 3201. 

Accordingly, the integral component calculation unit 
3203 calculates integral components K± (x s/ x e/ y S / y e ) based 
on the number of dimensions and the spatial resolution power 
5 (integral range) set by the conditions setting unit 3201, 
and the angle 0 of the data continuity information output 
from the data continuity detecting unit 101, and supplies 
the calculated result to the output pixel value calculation 
unit 3204 as an integral component table. 

10 The output pixel value calculation unit 32 04 calculates 

the right side of the above Expression (12 5) using the 
features table supplied from the features storage unit 3202, 
and the integral component table supplied from the integral 
component calculation unit 3203, and outputs the calculated 

15 result to the outside as the output pixel value M. 

Next, description will be made regarding image 
generating processing (processing in step S103 in Fig. 29) 
by the image generating unit 103 (Fig. 104) employing the 
two-dimensional reintegration method with reference to the 

20 flowchart in Fig. 105. 

For example, let us say that the light signals 
represented with the function F(x, y) shown in Fig. 102 have 
been cast in the sensor 2 so as to become an input image", 
and the actual world estimating unit 102 has already 

25 generated the approximation function f (x, y) for 
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approximating the function F(x, y) with one pixel 3231 such 
as shown in Fig. 106 as a pixel of interest at the 
processing in step S102 in Fig. 2 9 described above. 

Note that in Fig. 106, the pixel value (input pixel 
5 value) of the pixel 3231 is taken as P, and the shape of the 
pixel 3231 is taken as a square of which one side is 1 in 
length. Also, of the spatial directions, the direction in 
parallel with one side of the pixel 3231 is taken as the X 
direction, and the direction orthogonal to the X direction 
10 is taken as the Y direction. Further, a coordinates system 
(hereafter, referred to as a pixel-of -interest coordinates 
system) in the spatial directions (X direction and Y 
direction) of which the origin is the center of the pixel 
3231 is set. 

15 Also, let us say that in Fig. 106, the data continuity 

detecting unit 101, which takes the pixel 3231 as a pixel of 
interest, has already output the angle 6 as data continuity 
information corresponding to continuity of data represented 
with the gradient G f at the processing in step S101 in Fig. 

2 0 2 9 described above. 

Description will return to Fig. 105, and in this case, 
the conditions setting unit 3201 sets conditions (the number 
of dimensions and an integral range) at step S3201. 

For example , now, let us say that 5 has been set as the 

2 5 number of dimensions, and also spatial quadruple density 
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(spatial resolution power to cause the pitch width of a 
pixel to become half power in the upper/lower/left/right 
sides) has been set as an integral range. 

That is to say, in this case, it has been set that the 
5 four pixel 3241 through pixel 3244 are created newly in a 
range of -0.5 through 0.5 in the X direction, and also a 
range of -0.5 through 0.5 in the Y direction (in the range 
of the pixel 3231 in Fig. 106), such as shown in Fig. 107. 
Note that in Fig. 107 as well, the same pixel -of -interest 

10 coordinates system as that in Fig. 106 is shown. 

Also, in Fig. 107, M (1) represents the pixel value of 
the pixel 3241 to be generated from now, M (2) represents 
the pixel value of the pixel 3242 to be generated from now, 
M (3) represents the pixel value of the pixel 3243 to be 

15 generated from now, and M (4) represents the pixel value of 
the pixel 3244 to be generated from now. 

Description will return to Fig. 105, in step S3202, the 
features storage unit 3202 acquires the features of the 
approximation function f (x, y) supplied from the actual 

20 world estimating unit 102, and generates a features table. 
In this case, the coefficients w 0 through w 5 of the 
approximation function f (x) serving as a 5 -dimensional 
polynomial are supplied from the actual world estimating 
unit 102, and accordingly, (w 0 , wi , w 2 , w 3 , w 4 , w 5 ) is 

25 generated as a features table. 
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In step S3203, the integral component calculation unit 
3203 calculates integral components based on the conditions 
(the number of dimensions and an integral range) set by the 
conditions setting unit 3201, and the data continuity 
5 information (angle 0) supplied from the data continuity 
detecting unit 101, and generates an integral component 
table . 

Specifically, for example, let us say that numbers 
(hereafter, such a number is referred to as a mode number) 1 

10 through 4 are respectively appended to the pixel 3241 

through pixel 3244 to be generated from now, the integral 
component calculation unit 3203 calculates the integral 
components K± (x s/ x e , y S / y e ) of the above Expression (12 5) 
as a function of 1 (however, 1 represents a mode number) 

15 such as the integral components Ki (1) shown in the left 
side of the following Expression (127) . 
Ki (1) = Ki (x s , x e , y s , y e ) 

. . . (127) 

Specifically, in this case, the integral components K± 
20 (1) shown in the following Expression (128) are calculated. 





(1) 


= kj 


(-0. 5, 0, 0,0.5) 


ki 


(2) 


= kj 


(0, 0. 5, 0, 0.5) 


kj 


(3) 


= kj 


(-0. 5, 0, -0. 5,0) 


ki 


(4) 


= kj 


(0, 0. 5, -0. 5,0) 



. . . (128) 
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Note that in Expression (128) , the left side represents 
the integral components K± (1) , and the right side 
represents the integral components Ki (x s/ x e/ y S / Ye) ■ That 
is to say, in this case, 1 is any one of 1 thorough 4, and 
5 also i is any one of 0 through 5, and accordingly, 24 Ki (1) 
in total of 6 K± (1), 6 K± (2), 6 K± (3), and 6 K± (4) are 
calculated. 

More specifically, first, the integral component 
calculation unit 3203 calculates the variable s (s = cot 0) 

10 of the above Expression (122) using the angle 0 supplied 
from the data continuity detecting unit 101. 

Next, the integral component calculation unit 3203 
calculates the integral components K± (x s , x e , y S / y e ) of each 
right side of the four expressions in Expression (12 8) 

15 regarding i = 0 through 5 using the calculated variable s. 
Note that with this calculation of the integral components 
Ki (x s , x e , y S / Ye) / the above Expression (125) is employed. 

Subsequently, the integral component calculation unit 
3203 converts each of the 24 integral components Ki (x s , x e , 

2 0 y s , y e ) calculated into the corresponding integral components 
Ki (1) in accordance with Expression (128) , and generates an 
integral component table including the 24 integral 
components Ki (1) converted (i.e., 6 Ki (1), 6 Ki (2), 6 Ki 
(3) , and 6 K± (4) ) . 

25 Note that the sequence of the processing in step S3202 
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and the processing in step S32 03 is not restricted to the 
example in Fig. 105, the processing in step S32 03 may be 
executed first, or the processing in step S3202 and the 
processing in step S32 03 may be executed simultaneously. 

Next, in step S32 04, the output pixel value calculation 
unit 3204 calculates the output pixel values M (1) through M 

(4) respectively based on the features table generated by 
the features storage unit 3202 at the processing in step 
S3202, and the integral component table generated by the 
integral component calculation unit 32 03 at the processing 
in step S3203 . 

Specifically, in this case, the output pixel value 
calculation unit 3204 calculates each of the pixel value M 

(1) of the pixel 3241 (pixel of mode number 1) , the pixel 
value M (2) of the pixel 3242 (pixel of mode number 2) , the 
pixel value M (3) of the pixel 3243 (pixel of mode number 3) , 
and the pixel value M (4) of the pixel 3244 (pixel of mode 
number 4) shown in Fig. 107 by calculating the right sides 
of the following Expression (129) through Expression (132) 
corresponding to the above Expression (12 5) . 



n 



M(1) 



I WjXkjd) 



i=0 



. . . (129) 



n 



M(2) 



i=0 
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. . . (130) 



n 



M(3) 



I Wj xkj (3) 



i=0 



. . . (131) 



n 



M(4) 



1 WjXki(4) 



i=0 



. . . (132) 



However, in this case, each n of Expression (129) 
through Expression (132) becomes 5. 

In step S3205, the output pixel value calculation unit 
3204 determines regarding whether or not the processing of 
all the pixels has been completed. 

In step S3205, in the event that determination is made 
that the processing of all the pixels has not been completed, 
the processing returns to step S3202, wherein the subsequent 
processing is repeatedly performed. That is to say, the 
pixels that have not become a pixel of interest are 
sequentially taken as a pixel of interest, and the 
processing in step S3202 through S3204 is repeatedly 
performed . 

In the event that the processing of all the pixels has 
been completed (in step S3205, in the event that 
determination is made that the processing of all the pixels 
has been completed) , the output pixel value calculation unit 
3204 outputs the image in step S3206. Then, the image 
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generating processing ends. 

Thus, four pixels having higher spatial resolution than 
the input pixel 3231, i.e., the pixel 3241 through pixel 
3244 (Fig. 107) can be created by employing the two- 
5 dimensional reintegration method as a pixel at the pixel 
3231 of the input image (Fig. 106) from the sensor 2 (Fig. 
89) . Further, though not shown in the drawing, as described 
above, the image generating unit 103 can create a pixel 
having an arbitrary powered spatial resolution as to the 

10 input pixel 3231 without deterioration by appropriately 

changing an integral range, in addition to the pixel 3241 
through pixel 3244 . 

As described above, as description of the two- 
dimensional reintegration method, an example for subjecting 

15 the approximation function f (x, y) as to the spatial 

directions (X direction and Y direction) to two-dimensional 
integration has been employed, but the two-dimensional 
reintegration method can be applied to the time- space 
directions (X direction and t direction, or Y direction and 

20 t direction) . 

That is to say, the above example is an example in the 
case in which the light signals in the actual world 1 (Fig. 
89) have continuity in the spatial directions represented 
with the gradient G F such as shown in Fig. 102, and 

25 accordingly, an expression including two-dimensional 
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integration in the spatial directions (X direction and Y 
direction) such as shown in the above Expression (12 0) has 
been employed. However, the concept regarding two- 
dimensional integration can be applied not only to the 
5 spatial direction but also the time-space directions (X 

direction and t direction, or Y direction and t direction) . 

In other words, with the two-dimensional polynomial 
approximating method serving as an assumption of the two- 
dimensional reintegration method, it is possible to perform 

10 approximation using a two-dimensional approximation function 
f even in the case in which the image function F(x, y, t) 
representing the light signals has continuity in the time- 
space directions (however, X direction and t direction, or Y 
direction and t direction) as well as continuity in the 

15 spatial directions. 

Specifically, for example, in the event that there is 
an object moving horizontally in the X direction at uniform 
velocity, the direction of movement of the object is 
represented with like a gradient V F in the X-t plane such as 

20 shown in Fig. 108. In other words, it can be said that the 
gradient V F represents the direction of continuity in the 
time-space directions in the X-t plane. Accordingly, the 
data continuity detecting unit 101 (Fig. 89) can output 
movement 0 such as shown in Fig. 108 (strictly speaking, 

2 5 though not shown in the drawing, movement 0 is an angle 
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generated by the direction of data continuity represented 
with the gradient V f corresponding to the gradient V F and the 
X direction in the spatial direction) as data continuity 
information corresponding to the gradient V F representing 
continuity in the time-space directions in the X-t plane as 
well as the above angle 0 (data continuity information 
corresponding to the gradient G F representing continuity in 
the spatial directions in the X-Y plane) . 

Also, the actual world estimating unit 102 (Fig. 89) 
employing the two-dimensional polynomial approximating 
method can calculate the coefficients (features) vj± of an 
approximation function f (x, t) with the same method as the 
above method by employing the movement 0 instead of the 
angle 0 described above. However, in this case, the 
equation to be employed is not the above Expression (124) 
but the following Expression (133) . 

f (x, y) = I Wj (x-sxt) 

i=0 

. . . (133) 

Note that in Expression (133) , s is cot 0 (however, 0 is 
movement) . 

Accordingly, the image generating unit 103 (Fig. 89) 
employing the two-dimensional reintegration method can 
calculate the pixel value M by substituting the f (x, t) of 
the above Expression (133) for the right side of the 
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following Expression (134), and calculating this, 
t x 

M = G e x X *f *f (x.t)dxdt 

. . . (134) 

Note that in Expression (134) , t s represents an 
integration start position in the t direction, and t e 
represents an integration end position in the t direction. 
Similarly, x s represents an integration start position in 
the X direction, and x e represents an integration end 
position in the X direction. G e represents a predetermined 
gain. 

Alternately, an approximation function f (y, t) focusing 
attention on the spatial direction Y instead of the spatial 
direction X can be handled as the same way as the above 
approximation function f (x, t). 

Incidentally, in Expression (133) , it becomes possible 
to obtain data not integrated in the temporal direction, 
i.e., data without movement blurring by regarding the t 
direction as constant, i.e., by performing integration while 
ignoring integration in the t direction. In other words, 
this method may be regarded as one of two-dimensional 
reintegration methods in that reintegration is performed on 
condition that one certain dimension of the two-dimensional 
approximation function f is constant, or in fact, may be 
regarded as one of one-dimensional reintegration methods in 
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that one-dimensional reintegration in the X direction is 
performed. 

Also, in Expression (134) , an integral range may be set 
arbitrarily, and accordingly, with the two-dimensional 
5 reintegration method, it becomes possible to create a pixel 
having an arbitrary powered resolution as to the original 
pixel (pixel of an input image from the sensor 2 (Fig. 89)) 
without deterioration by appropriately changing this 
integral range . 

10 That is to say, with the two-dimensional reintegration 

method, it becomes possible to create temporal resolution by 
appropriately changing an integral range in the temporal 
direction t. Also, it becomes possible to create spatial 
resolution by appropriately changing an integral range in 

15 the spatial direction X (or spatial direction Y) . Further, 
it becomes possible to create both temporal resolution and 
spatial resolution by appropriately changing each integral 
range in the temporal direction t and in the spatial 
direction X. 

2 0 Note that as described above, creation of any one of 

temporal resolution and spatial resolution may be performed 
even with the one-dimensional reintegration method, but 
creation of both temporal resolution and spatial resolution 
cannot be performed with the one-dimensional reintegration 

25 method in theory, which becomes possible only by performing 
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two-dimensional or more reintegration. That is to say, 
creation of both temporal resolution and spatial resolution 
becomes possible only by employing the two-dimensional 
reintegration method and a later-described three-dimensional 
5 reintegration method. 

Also, the two-dimensional reintegration method takes 
not one-dimensional but two-dimensional integration effects 
into consideration, and accordingly, an image more similar 
to the light signal in the actual world 1 (Fig. 89) may be 
10 created. 

Next, description will be made regarding a three- 
dimensional reintegration method with reference to Fig. 109 
and Fig. 110. 



15 approximation function f (x, y, t) has been created using the 
three-dimensional approximating method, which is an 
assumption . 

In this case, with the three-dimensional reintegration 
method, the output pixel value M is calculated as the 
20 following Expression (135) . 



With the three-dimensional reintegration method, the 




. . . (135) 



Note that in Expression (135) , t s represents an 



integration start position in the t direction, and t e 
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represents an integration end position in the t direction. 
Similarly, y s represents an integration start position in 
the Y direction, and y e represents an integration end 
position in the Y direction. Also, x s represents an 
5 integration start position in the X direction, and x e 

represents an integration end position in the X direction. 
G e represents a predetermined gain. 

Also, in Expression (135) , an integral range may be set 
arbitrarily, and accordingly, with the three-dimensional 

10 reintegration method, it becomes possible to create a pixel 
having an arbitrary powered time-space resolution as to the 
original pixel (pixel of an input image from the sensor 2 
(Fig. 89) ) without deterioration by appropriately changing 
this integral range. That is to say, upon the integral 

15 range in the spatial direction being reduced, a pixel pitch 
can be reduced without restraint. On the other hand, upon 
the integral range in the spatial direction being enlarged, 
a pixel pitch can be enlarged without restraint. Also, upon 
the integral range in the temporal direction being reduced, 

20 temporal resolution can be created based on an actual 
waveform. 

Fig. 109 represents a configuration example of the 
image generating unit 103 employing the three-dimensional 
reintegration method. 
25 As shown in Fig. 109, this example of the image 
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generating unit 103 includes a conditions setting unit 3301, 
features storage unit 33 02, integral component calculation 
unit 3303, and output pixel value calculation unit 3304. 
The conditions setting unit 33 01 sets the number of 
5 dimensions n of the approximation function f (x, y, t) based 
on the actual world estimating information (with the example 
in Fig. 109, features of the approximation function f (x, y, 
t) ) supplied from the actual world estimating unit 102. 

The conditions setting unit 33 01 sets an integral range 

10 in the case of reintegrating the approximation function f (x, 
y, t) (in the case of calculating output pixel values) . 
Note that an integral range set by the conditions setting 
unit 33 01 needs not to be the width (vertical width and 
horizontal width) of a pixel or shutter time itself. For 

15 example, it becomes possible to determine a specific 

integral range in the spatial direction as long as the 
relative size (spatial resolution power) of an output pixel 
(pixel to be generated from now by the image generating unit 
103) as to the spatial size of each pixel of an input image 

20 from the sensor 2 (Fig. 89) is known. Similarly, it becomes 
possible to determine a specific integral range in the 
temporal direction as long as the relative time (temporal 
resolution power) of an output pixel value as to the shutter 
time of the sensor 2 (Fig. 89) is known. Accordingly, the 

25 conditions setting unit 3301 can set, for example, a spatial 



- 323 - 

S04P0850 

resolution power and temporal resolution power as an 
integral range . 

The features storage unit 3302 temporally stores the 
features of the approximation function f (x, y, t) 
5 sequentially supplied from the actual world estimating unit 
102. Subsequently, upon the features storage unit 3302 
storing all of the features of the approximation function 
f (x, y, t) , the features storage unit 3302 generates a 
features table including all of the features of the 
10 approximation function f (x, y, t) , and supplies this to the 
output pixel value calculation unit 3304. 

Incidentally, upon the right side of the approximation 
function f (x, y) of the right side of the above Expression 
(13 5) being expanded (calculated) , the output pixel value M 
15 is represented as the following Expression (136) . 
n 

M= I WiXkj(x Sf x e , y 5 , y e , t s , t e ) 

i=0 

. . . (136) 

In Expression (136) , K± (x s , x e , y s/ y e , t s , t e ) 
represents the integral components of the i -dimensional term. 
20 However, x s represents an integration range start position 
in the X direction, x e represents an integration range end 
position in the X direction, y s represents an integration 
range start position in the Y direction, y e represents an 
integration range end position in the Y direction, t s 
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represents an integration range start position in the t 
direction, and t e represents an integration range end 
position in the t direction, respectively. 

The integral component calculation unit 33 03 calculates 
5 the integral components Ki (x S/ x e/ y S / Ye/ t s , t e ) . 

Specifically, the integral component calculation unit 
3303 calculates the integral components K± (x s , x e/ y S / y e / t S/ 
t e ) based on the number of dimensions and the integral range 
(spatial resolution power or temporal resolution power) set 

10 by the conditions setting unit 3301, and the angle 0 or 

movement 0 of the data continuity information output from 
the data continuity detecting unit 101, and supplies the 
calculated results to the output pixel value calculation 
unit 3304 as an integral component table. 

15 The output pixel value calculation unit 3304 calculates 

the right side of the above Expression (13 6) using the 
features table supplied from the features storage unit 3302, 
and the integral component table supplied from the integral 
component calculation unit 3303, and outputs the calculated 

2 0 result to the outside as the output pixel value M. 

Next, description will be made regarding image 
generating processing (processing in step S103 in Fig. 29) 
by the image generating unit 103 (Fig. 109) employing the 
three-dimensional reintegration method with reference to the 

2 5 flowchart in Fig. 110. 



- 325 - 

S04P0850 

For example, let us say that the actual world 
estimating unit 102 (Fig. 89) has already generated an 
approximation function f (x, y, t) for approximating the 
light signals in the actual world 1 (Fig. 89) with a 
5 predetermined pixel of an input image as a pixel of interest 
at the processing in step S102 in Fig. 29 described above. 

Also, let us say that the data continuity detecting 
unit 101 (Fig. 89) has already output the angle 0 or 
movement 6 as data continuity information with the same 

10 pixel as the actual world estimating unit 102 as a pixel of 
interest at the processing in step S101 in Fig. 29. 

In this case, the conditions setting unit 3301 sets 
conditions (the number of dimensions and an integral range) 
at step S3301 in Fig. 110. 

15 In step S3302, the features storage unit 3302 acquires 

the features wi of the approximation function f (x, y, t) 
supplied from the actual world estimating unit 102, and 
generates a features table. 

In step S33 03, the integral component calculation unit 

20 3303 calculates integral components based on the conditions 
(the number of dimensions and an integral range) set by the 
conditions setting unit 3301, and the data continuity 
information (angle 0 or movement 0) supplied from the data 
continuity detecting unit 101, and generates an integral 

25 component table. 
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Note that the sequence of the processing in step S33 02 
and the processing in step S33 03 is not restricted to the 
example in Fig. 110, the processing in step S3303 may be 
executed first, or the processing in step S3302 and the 
5 processing in step S3 3 03 may be executed simultaneously. 

Next, in step S3304, the output pixel value calculation 
unit 33 04 calculates each output pixel value based on the 
features table generated by the features storage unit 33 02 
at the processing in step S33 02, and the integral component 
10 table generated by the integral component calculation unit 

3303 at the processing in step S3303 . 

In step S33 05, the output pixel value calculation unit 

3304 determines regarding whether or not the processing of 
all the pixels has been completed. 

15 In step S3305, in the event that determination is made 

that the processing of all the pixels has not been completed, 
the processing returns to step S3302, wherein the subsequent 
processing is repeatedly performed. That is to say, the 
pixels that have not become a pixel of interest are 

20 sequentially taken as a pixel of interest, and the 

processing in step S3302 through S3304 is repeatedly 
performed. 

In the event that the processing of all the pixels has 
been completed (in step S3305, in the event that 
25 determination is made that the processing of all the pixels 
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has been completed) , the output pixel value calculation unit 
3304 outputs the image in step S3306. Then, the image 
generating processing ends. 

Thus, in the above Expression (13 5) , an integral range 
5 may be set arbitrarily, and accordingly, with the three- 
dimensional reintegration method, it becomes possible to 
create a pixel having an arbitrary powered resolution as to 
the original pixel (pixel of an input image from the sensor 
2 (Fig. 89)) without deterioration by appropriately changing 

10 this integral range. 

That is to say, with the three-dimensional 
reintegration method, appropriately changing an integral 
range in the temporal direction enables temporal resolution 
to be created. Also, appropriately changing an integral 

15 range in the spatial direction enables spatial resolution to 
be created. Further, appropriately changing each integral 
range in the temporal direction and in the spatial direction 
enables both temporal resolution and spatial resolution to 
be created. 

20 Specifically, with the three-dimensional reintegration 

method, approximation is not necessary when degenerating 
three dimension to two dimension or one dimension, thereby 
enabling high-precision processing. Also, movement in an 
oblique direction may be processed without degenerating to 

25 two dimension. Further, no degenerating to two dimension 
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enables process at each dimension. For example, with the 
two-dimensional reintegration method, in the event of 
degenerating in the spatial directions (X direction and Y 
direction) , process in the t direction serving as the 
5 temporal direction cannot be performed. On the other hand, 
with the thee-dimensional reintegration method, any process 
in the time- space directions may be performed. 

Note that as described above, creation of any one of 
temporal resolution and spatial resolution may be performed 

10 even with the one-dimensional reintegration method, but 

creation of both temporal resolution and spatial resolution 
cannot be performed with the one-dimensional reintegration 
method in theory, which becomes possible only by performing 
two-dimensional or more reintegration. That is to say, 

15 creation of both temporal resolution and spatial resolution 
becomes possible only by employing the above two-dimensional 
reintegration method and the three-dimensional reintegration 
method . 

Also, the three-dimensional reintegration method takes 
2 0 not one -dimensional and two-dimensional but three- 
dimensional integration effects into consideration, and 
accordingly, an image more similar to the light signal in 
the actual world 1 (Fig. 89) may be created. 

Next, with the signal processing device 4 shown in Fig. 
25 3, data continuity is detected at the data continuity 
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detecting unit 101, and estimation of actual world 1 signal 
waveforms, i.e., for example, an approximation function 
approximating the X-cross-section waveform F (x) , is obtained 
at the actual world estimating unit 102, based on the 
5 continuity. 

Thus, waveform estimation of actual world 1 signals is 
performed at the signal processing device 4 based on 
continuity, so in the event that the continuity detected by 
the data continuity detecting unit 101 is incorrect or 
10 detection precision thereof is poor, the estimation 

precision of the waveform of the actual world 1 signals is 
also poor. 

Also, the signal processing device 4 performs signal 
processing based on continuity which actual world 1 signals, 

15 which is an image in this case, for example, have, signal 

processing with better precision than signal processing with 
other signal processing devices can be performed for the 
parts of the actual world 1 signals where continuity exists, 
and consequently, an image closer to the image corresponding 

20 to the actual world 1 signals can be output. 

However, the signal processing device 4 cannot perform 
signal processing for parts of the actual world 1 signals 
where no clear continuity exists at the same level of 
precision as four parts were continuity does exist, since 

25 the signal processing is being performed based on continuity, 
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and consequently, an image including error with regard to 
the image corresponding to the actual world 1 signals is 
output . 

Accordingly, in order to obtain an image closer to the 
5 image corresponding to the actual world 1 signals with the 
signal processing device 4, processing regions where signal 
processing with the signal processing device 4 is to be 
performed, precision of the continuity used with the signal 
processing device 4, and so forth, are issues. 
10 Here, Fig. Ill illustrates the configuration example of 

another embodiment of the signal processing device 4 in Fig. 
1 . 

In Fig. Ill, the signal processing device 4 comprises a 
processing region setting unit 10001, a continuity setting 

15 unit 10002, an actual world estimating unit 10003, an image 
generating unit 10004, an image display unit 10005, and a 
user I/F (Interface) 10006. 

With the signal processing device 4 of which the 
configuration is shown in Fig. Ill, image data (input image) 

20 which is an example of the data 3, is input from the sensor 
2 (Fig. 1), and the input image is supplied to the 
processing region setting unit 10001, the continuity setting 
unit 10002, the actual world estimating unit 10003, the 
image generating unit 10004, and the image display unit 

25 10005. 



- 331 - 

S04P0850 

The processing region setting unit 10001 sets the 
processing region for the input image, and supplies 
processing region information identifying the processing 
region to the continuity setting unit 10002, the actual 

{ 

5 world estimating unit 10003, and the image generating unit 

r 

10004. 

The continuity setting unit 10002 recognizes the 
processing region in the input image from the processing 
region information supplied from the processing regions 

10 setting unit 10001, sets the continuity of the actual world 

1 signals lost from the image data at that processing region, 
and supplies continuity information representing that 
continuity to the actual world estimating unit 10003 and the 
image generating unit 10004. 

15 The actual world estimating unit 10003 is configured of 

a model generating unit 10011, equation generating unit 
10012, and actual world waveform estimating unit 10013, and 
estimates the actual world 1 signals from the image data 
within the processing region, according to the continuity of 

2 0 the corresponding actual world 1 signals. 

That is to say, the model generating unit 10011 
recognizes the processing region in the input image from the 
processing region information supplied from the processing 
region setting unit 10001, generates a function serving as a 

25 model which models the relation between the pixel values of 
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the pixels within the processing region and the actual world 
1 signals, according to the pixels making up the processing 
region and the continuity of the actual world 1 signals 
corresponding to the image data in the processing region, 
5 and supplies this to the equation generating unit 10012. 

The question generating unit 10012 recognizes the 
processing region in the input image from the processing 
region information supplied from the processing regions 
setting unit 10001. Further, the equation generating unit 

10 10012 substitutes the pixel values of each of the pixels 

making up the processing region into the function serving as 
a model that has been supplied from the model generating 
unit 10011, thereby generating an equation, which is 
supplied to the actual world waveform estimating unit 10013. 

15 The actual world waveform estimating unit 10013 

estimates the waveform of the actual world 1 signals, by 
computing the equation supplied from the equation generating 
unit 10012. That is to say, the actual world waveform 
estimating unit 10013 obtains an approximation function 

20 approximating the actual world 1 signals by solving the 

equation supplied from the equation generating unit 10012, 
and supplies the approximation function to the image 
generating unit 10004, as estimation results of the waveform 
of the actual world 1 signals. Note that approximation 

25 function approximating the actual world 1 signals include 
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functions with constant function values, regardless of 
argument value . 

The image generating unit 10004 generates signals 
closer approximating the actual world 1 signals, based on 
5 the approximation function representing the waveform of the 
actual world 1 signals estimated at the actual world 
estimating unit 10003, and the continuity information 
supplied from the continuity setting unit 10002. That is to 
say, the image generating unit 10004 recognizes the . 

10 processing region in the input image from the processing 
region information supplied from the processing region 
setting unit 10001, and generates image data closer 
approximating the image corresponding to the actual world 1 
signals with regard to the processing region, based on the 

15 approximation function supplied from (the actual world 
waveform estimating unit 10013) of the actual world 
estimating unit 10003, and the continuity information 
supplied from the continuity setting unit 10002. 

Further, the image generating unit 10004 synthesizes 

2 0 the input image and image data generated based on the 
approximation function (hereafter referred to as 
approximation image as appropriate) , generates an image 
wherein the portion of the processing region of the input 
image has been substituted with the approximation image, and 

2 5 supplies the image as an output image to the image display 
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unit 10005. 

The image display unit 10005 is configured of a CRT 
(Cathode Ray Tube) or an LCD (Liquid Crystal Display) , and 
displays input images, or output images supplied from the 
5 image generating unit 10004. 

Note that the image display unit 10005 can be 
configured of single or multiple CRTs or LCDs. In the event 
of configuring the image display unit 10005 with a single 
CRT or LCD, an arrangement maybe made wherein the screen of 

10 the single CRT or LCD is divided into multiple screens, with 
input images displayed on one screen and output images 
displayed on another screen. Further, in the event of 
configuring the image display unit 10005 of multiple CRTs or 
LCDs, an arrangement may be made wherein input images are 

15 displayed on one CRT or LCD, and outputs images are 
displayed on another CRT or LCD. 

Also, the image display unit 10005 performs display of 
various types in accordance with the output of the user I/F 
10006. That is to say, the image display unit 10005 

2 0 displays a cursor, for example, and in the event that the 
user operates the user I/F 10006 so as to move the cursor, 
the cursor is moved in accordance with the operations 
thereof. Also, in the event that the user operates the user 
I/F 10006 so as to select a predetermined range, the image 

25 display unit 10005 displays a frame surrounding the range 
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selected on the screen in accordance with the operations 
thereof . 

The user I/F 10006 is operated by the user, and in 
accordance with the user operations, supplies information 
5 relating to at least one of, for example, processing range, 
continuity, and real world signals, to the processing region 
setting unit 10001, continuity setting unit 10002, or actual 
world estimating unit 10003. 

That is to say, the user views an input image or output 

10 image displayed on the image display unit 10005, and 

operates the user I/F 10006 so as to provide input with 
regard to the input image or output image. The user I/F 
10006 is operated by the user, and in accordance with the 
user operations, supplies information relating to processing 

15 range, continuity, or real world signals, to the processing 
region setting unit 10001, continuity setting unit 10002, or 
actual world estimating unit 10003, as assisting information 
for assisting the processing of the processing region 
setting unit 10001, continuity setting unit 10002, or actual 

20 world estimating unit 10003. 

Upon assisting information being supplied from the user 
I/F 10006, the processing region setting unit 10001, 
continuity setting unit 10002, or actual world estimating 
unit 10003 each perform setting of processing region, 

25 setting of continuity, or estimation of actual world 1 
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signals, based on the assisting information. 

Note however, that the processing region setting unit 
10001, continuity setting unit 10002, or actual world 
estimating unit 10003, can each perform the setting of 
5 processing region, setting of continuity, or estimation of 
actual world 1 signals, even without using the assisting 
information, i.e., even without the user operating the user 
I/F 10006. 

Specifically, with the processing region setting unit 

10 10001, processing can be performed in the same way as with 
the data continuity detecting unit 101 shown in Fig. 3, as 
described with reference to Fig. 30 to Fig. 48, wherein a 
continuity region is detected from the input image, and a 
rectangular (oblong) region surrounding the continuity 

15 region is set as a processing region. 

Also, with the continuity setting unit 10002, 
processing can be performed in the same way as with the data 
continuity detecting unit 101 shown in Fig. 3, as described 
with reference to Fig. 49 through Fig. 57, wherein data 

2 0 continuity is detected from the input image, and the 

continuity of the corresponding actual world 1 signals is 
set based on the continuity of that data, i.e., for example, 
the data continuity is set as continuity of the actual world 
1 signals with no change. 

25 Further, with the actual world estimating unit 10003, 



1 
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processing may be performed in the same way as with the 
actual world estimating unit 102 shown in Fig. 3, as 
described with reference to Fig. 58 through Fig. 88, where 
actual world 1 signals are estimated from the image data of 
5 the processing region set by the processing region setting 
unit 10001, corresponding to the continuity set by the 
continuity setting unit 10002 . Note that while the data 
continuity has been used for estimating actual world 1 
signals that the actual world estimating unit 102 in Fig. 3, 

10 an arrangement may be made wherein the continuity of 

corresponding actual world 1 signals is used instead of the 
data continuity for estimating actual world 1 signals. 

Next, the processing of the signal processing device 4 
shown in Fig. Ill will be described with reference to the 

15 flowchart in Fig. 112. 

First, in step S10001, the signal processing device 4 
performs pre-processing, and the flow proceeds to step 
S10002. That is to say, the signal processing device 4 
supplies one frame or one field, for example, of the input 

2 0 image, supplied from the sensor 2 (Fig. 1) as data 3, to the 
processing region setting unit 10001, continuity setting 
unit 10002, actual world estimating unit 10003, image 
generating unit 10004, and image display unit 10005. 
Further, the signal processing unit 4 causes the image 

25 display unit 10005 to display the input image. 
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In step S10002, the user I/F 10006 determines whether 
or not there has been some sort of user input, by the user 
operating the user I/F 10006. In step S10002, in the event 
that determination is made that there is no user input, i.e., 
5 in the event that the user has made no operations, step 
S10003 through step S10005 are skipped, and the flow 
proceeds to step S10006. 

Also, in the event that determination is made in step 

510002 that there has been user input, i.e., in the event 
10 that the user has viewed the input image displayed on the 

image display unit 10005 and operated the user I/F 10006, 
thereby making user input indicating some sort of 
instruction or information, the flow proceeds to step S10003, 
where the user I/F 10006 determines whether or not the user 

15 input is ending instructions for instructing ending of the 
processing of the signal processing device 4 . 

In the event that determination is made in step S10003 
that the user input is ending instructions, the signal 
processing device 4 ends processing. 

20 Also, in the event that determination is made in step 

510003 that the user input is not ending instructions, the 
flow proceeds to step S10004, where the user I/F 10006 
determines whether or not the user input is assisting 
information. In the event that determination is made in 

25 step S10004 that the user input is not assisting information, 
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the flow skips step S10005, and proceeds to step S10006. 

Also, in the event that determination is made in step 
S10004 that the user input is assisting information, the 
flow proceeds to step S10005, where the user I/F 10006 
5 supplies the assisting information to the processing region 
setting unit 10001, continuity setting unit 10002, or actual 
world estimating unit 10006, and the flow proceeds to step 
S10006. 

In step S10006, the processing region setting unit 
10 10001 sets the processing region based on the input image, 
and supplies the processing region information identifying 
the processing region to the continuity setting unit 10002, 
actual world estimating unit 10003, and image generating 
unit 10004, and the flow proceeds to step S10007. Now, in 
15 the event that assisting information has been supplied from 
the user I / F 10006 in that immediately-preceding step 
S10005, the processing region setting unit 10001 performs 
setting of the processing region using that assisting 
information. 

20 In step S10007, the continuity setting unit 10002 

recognizes the processing region in the input image, from 
the processing region information supplied from the 
processing region setting unit 10001. Further, the 
continuity setting unit 10002 sets continuity of the actual 

25 world 1 signals that has been lost in the image data of the 
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processing region, and supplies continuity information 
indicating the continuity thereof to the actual world 
estimating unit 10003, and the flow proceeds to step S10008. 
Now, in the event that assisting information has been 
5 supplied from the user I / F 10006 in that immediately- 
preceding step S10005, the continuity setting unit 10002 
performs setting of continuity using that assisting 
information . 

In step S10008, the actual world estimating unit 10003 
10 estimates actual world 1 signals regarding to the image data 
with in the processing region of the input image, according 
to the continuity of the corresponding actual world 1 
signals . 

That is to say, at the actual world estimating unit 
15 10003, the model generating unit 10011 recognizes the 

processing region in the input image, from the processing 
region information supplied from the processing region 
setting unit 10001, and also recognizes continuity of the 
actual world 1 signals corresponding to the image data in 
20 the processing region, from the continuity information 

supplied from the continuity setting unit 10002. Further 
the model generating unit 10011 generates a function serving 
as a model modeling the relation between the pixel values of 
each of the pixels within the processing region and the 
2 5 actual world 1 signals, according to the pixels making up 
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the processing region iri the input image, and the continuity 
of actual world 1 signals corresponding to the image data of 
the processing region, and supplies this to the equation 
generating unit 10012. 
5 The equation generating unit 10012 recognizes the 

processing region in the input image from the processing 
region information supplied from the processing region 
setting unit 10001, and substitutes the pixel values of each 
of the pixels of the input image making up the processing 

10 region into the function serving as the model which is 
supplied from the model generating unit 10 011, thereby- 
generating an equation for obtaining an approximation 
function approximating the actual world 1 signals, which is 
supplied to the actual world waveform estimating unit 10013. 

15 The actual world waveform estimating unit 10013 

estimates the waveform of the actual world 1 signals by 
computing the equation supplied from the equation generating 
unit 10012. That is, the actual world waveform estimating 
unit 10013 obtains the approximation function serving as a 

2 0 model modeling the actual world 1 signals by solving the 

equation supplied from the equation generating unit 10012, 
and supplies the approximation function to the image 
generating unit 10004 as estimation results of the waveform 
of the actual world 1 signals. 

25 Note that with the actual world estimating unit 10003, 
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in the event that assisting information has been supplied 
from the user I/F 10006 in the immediately-preceding step 
S10005, at the model generating unit 10011 and equation 
generating unit 10012, processing is performed using that 
5 assisting information. 

Following processing of step S10008, the flow proceeds 
to step S10009, where the image generating unit 10004 
generates signals closer approximating the actual world 1 
signals based on the approximation function approximating 

10 the waveform of the actual world 1 signals supplied from 
(the actual world waveform estimating unit 10013 of) the 
actual world estimating unit 10003. That is to say, the 
image generating unit 10004 recognizes the processing region 
in the input image from the processing region information 

15 supplied from the processing region setting unit 10001, and 
with regard to this processing region, generates an 
approximation image which is image data closer approximating 
the image corresponding to the actual world 1 signals, based 
on the approximation function supplied from the actual world 

20 estimating unit 10003. Further the image generating unit 
10004 generates, as an output image, an image wherein the 
portion of the processing region of the input image has been 
replaced with the approximation image, and supplies this to 
the image display 100 05, and the flow proceeds from step 

25 S10009 to step S10010. 
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In step S10010, the image display unit 10005 displays 
the output image supplied from the image generating unit 
10004 instead of the input image displayed in step S10001, 
or along with the input image, and the flow proceeds to step 
5 S10011. 

In step S10011, the user I/F 10006 determines whether 
or not there has been user input of some sort by the user 
operating the user I/F 10006, in the same way as with step 
S10002, and in the event that determination is made that 

10 there has been no user input, i.e., in the event that the 

user has made no operations, the flow returns to step S10011, 
and awaits user input. 

Also, in the event that determination is made in step 
S10011 that there has been user input, i.e., in the event 

15 that the user has viewed the input image or output image 

displayed on the image display unit 10005 and operated the 
user I/F 10006, thereby making some sort of user input 
indicating instruction or information, the flow proceeds to 
step S10012, where the user I/F 10006 determines whether or 

2 0 not the user input is ending instructions instructing ending 
of the processing of the signal processing device 4 . 

In the event that determination is made in step S10012 
that the user input is ending instructions, the signal 
processing device 4 ends processing. 

2 5 Also, in the event that determination is made in step 
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S10012 that the user input is not ending instructions, the 
flow proceeds to step S10013, where the user I/F 10006 
determines whether or not the user input is assisting 
information. In the event that determination is made in 
5 step S10013 that the user input is not assisting information, 
the flow returns to step S10011, and the same processing is 
repeated thereafter . 

Also, in step S10013, in the event that determination 
is made that the user input is assisting information, the 

10 flow returns to step S10005, and as described above, the 
user I/F 10006 supplies the assisting information to the 
processing region setting unit 10001, the continuity setting 
unit 10002, or the actual world estimating unit 10006. The 
flow then proceeds from step S10005 to step S10006, and 

15 hereafter the same processing is repeated. 

As described above, with the signal processing device 4 
shown in Fig. Ill, assisting information for assisting the 
processing region setting unit 10001, the continuity setting 
unit 10002, or the actual world estimating unit 10006, is 

20 supplied from the user I/F 10006 to the processing region 
setting unit 10001, the continuity setting unit 10002, or 
the actual world estimating unit 10003 and processing region 
setting, continuity setting, or actual world 1 signals 
estimation is performed at the processing region setting 

25 unit 10001, the continuity setting unit 10002, or the actual 
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world estimating unit 10003 based on the assisting 
information from the user i/F 10006, thereby improving the 
processing precision of the processing region setting unit 
10001, the continuity setting unit 10002, or the actual 
5 world estimating unit 10003, and enabling, for example, a 
high-quality output image meeting user preferences. 

Next, various types of application examples of the 
signal processing device 4 shown in Fig. Ill will be 
described. 

10 Fig. 113 illustrates a configuration example of an 

embodiment of an application of the signal processing device 
4 shown in Fig. 111. 

An image of, for example, one frame or one field as the 
data 3 from the sensor 2 is input into the signal processing 

15 device 4 in Fig. 113. Here, let us say that an object 

having a predetermined shape is displayed in the input image, 
such object moving in the horizontal direction (crosswise 
direction) of the input image at a constant speed of v 
pixels per shutter time (exposure time) . In other words, 

2 0 regarding the input image, the object is moving in the 

horizontal direction at the movement amount of v pixels, and 
therefore because of the time integration effect of the 
sensor 2, the light signal of the object and the light 
signal of the portion other than the object are mixed (time 

2 5 mixture), and thus, the image is blurred at portions such as 
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the borders of the object. With the signal processign 
device 4 shownh in Fig. 113, a high-quality outptu image is 
generated, wherein movement blurring due to such time 
mixture has been elimited from the input image. 
5 In Fig. 113, the processing region setting unit 15011, 

the continuity setting unit 15012, the actual world 
estimating unit 15013, image generating unit 15014, image 
display unit 15015, and user I/F 15016, each correspond to 
the processing region setting unit 10001, the continuity 

10 setting unit 10002, the actual world estimating unit 10003, 
image generating unit 10004, image display unit 10005, and 
user I/F 10006 shown in Fig. Ill, and basically perform the 
same processing as the processing region setting unit 10001, 
the continuity setting unit 10002, the actual world 

15 estimating unit 10003, image generating unit 10004, image 
display unit 10005, and user I/F 10006. Further, in Fig. 
113, the actual world estimating unit 15013 comprises a 
model generating unit 15021, an equation generating unit 
15022, and actual world waveform estimating unit 15023. The 

2 0 model generating unit 15021, equation generating unit 15022, 
and actual world waveform estimating unit 15023 correspond 
to each of the model generating unit 10011, equation 
generating unit 10012, and actual world waveform estimating 
unit 10013 in Fig. Ill, and basically perform the same 

25 functions as each of the model generating unit 10011, 
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equation generating unit 10012, and actual world waveform 
estimating unit 10013 . 

Note however, that in Fig. 113, the assisting 
information which the user I/F 15016 outputs by the user 
operating the user I/F 15016, is supplied only to the 
equation generating unit 15022. 

In other words, in Fig. 113, by operating the user I/F 
15016, with the equation generating unit 15022, the user can 
set the constraint conditions which constrain the relation 
between each of the pixels of an image with no movement 
blurring, which is an image of the actual world 1 
corresponding to the input image, and when the user performs 
an operation to set the constraint conditions, the user I/F 
15016 supplies the constraint condition information 
expressing the constraint conditions set by the operation to 
the equation generating unit 15022 . 

Specifically, for example, a user who views an image 
displayed on the image display unit 15015 reflecting a 
blurring image guesses the original image, that is to say, 
an image without movement blurring which is an image of the 
actual world 1 corresponding to the input image. Also, for 
example, the user guesses predetermined regions where 
blurring is occurring as the edge portions, and guesses the 
relation of the pixel values between each of the pixels (the 
constraint conditions) , such as the difference being large 
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between the pixel values (level) between the adjacent pixels 
at the edge portions. Further, the user can set the 
conditions suitable for the guessed relation of the pixels 
value between each of the pixels into the equation 
5 generating unit 15022, by operating the user I/F 15016. 

Also, in Fig. 113, the processing region information, 
which is supplied from the processing region setting unit 
15011 to the actual world estimating unit 15013, is supplied 
to the model generating unit 15021 and the equation 

10 generating unit 15022 of the actual world estimating unit 
15013. The model generating unit 15021 and the equation 
generating unit 15022 recognize the processing region in the 
input image from the processing region information which is 
supplied from the processing region setting unit 15011. 

15 The continuity setting unit 15012 sets a movement 

amount (movement vector) in the input image as the 
continuity information that an object is moving in the 
horizontal direction of the image at the constant speed of v 
pixels per shutter time, and supplies this movement amount v 

20 as the continuity information to the model generating unit 
15021 of the actual world estimating unit 15013. 

The model generating unit 15021 generates an equation 
(hereafter referred to as model equation as appropriate) as 
a model (hereafter referred to as relation model as 

25 appropriate) , modeling the relation between the signal of 
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the actual world 1 and the pixel values of each of the 
pixels of the movement blurring image as the input image 
which considers the movement amount v as the continuity 
information supplied from the continuity setting unit 15012, 
regarding the processing region recognized by the processing 
region information which is supplied from the processing 
region setting unit 15011 , and supplies the generated model 
equation to the equation generating unit 15022. 

The equation generating unit 15022 generates a 
constraint condition Expression (hereafter referred to as 
constraint condition expression as appropriate) , based on 
the constraint condition information supplied from the user 
I/F 15016, and substitutes the pixel values of each of the 
pixels of the input image comprising the processing region 
into this constraint condition expression and the model 
equation which is supplied from the model generating unit 
15021 to the equation generating unit 15022, and thus, 
generates an equation to find the approximation function as 
a model (hereafter referred to as approximation model) 
wherein the signal of the actual world 1 is modeled, and 
supplies this to the actual world waveform estimating unit 
15023 . 

The actual world waveform estimating unit 15023 
estimates the waveform of the signal of the actual world 1, 
by computing the equation supplied from the equation 
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generating unit 15022. In other words, the actual world 
waveform estimating unit 15023 finds the approximation 
function as an approximation model, and supplies the 
approximation function to the image generating unit 15014 as 
5 the estimation result of the signal waveform of the actual 
world 1. Here, the approximation function which 
approximates the signal of the actual world 1 has contained 
a function value that is a fixed function, regardless of the 
argument value . 

10 The image generating unit 15014 recognizes the 

processing region of the input image, based on the 
processing region information supplied from the processing 
region setting unit 15011. Also, the image generating unit 
15014 generates a signal more closely approximating the 

15 signal of the actual world 1, from the approximation 

function supplied from the actual world waveform estimating 
unit 15023, that is to say, generates an image with no 
movement blurring, and replaces the image of the processing 
region of the input image with an image wherein the image 

2 0 has no movement blurring, generates this as an output image, 
and supplies this to the image display unit 15015. 

Next, the processing of the signal processing device 4 
in Fig. 113 will be described with reference to the 
flowchart in Fig. 114. 

25 First, in step S15001, the signal processing device 4 
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performs pre-processing, and the flow proceeds to step 
S15002. That is to say, the signal processing device 4 
supplies one frame or one field, for example, of the input 
image, supplied from the sensor 2 (Fig. 1) as data 3, to the 
5 processing region setting unit 15011, continuity setting 
unit 15012, actual world estimating unit 15013, image 
generating unit 15014, and image display unit 15015. 
Further, the signal processing unit 4 causes the image 
display unit 15015 to display the input image. 

10 In step S15002, the user I/F 15016 determines whether 

or not there has been some sort of user input, by the user 
operating the user I/F 15016. In step S15002, in the event 
that determination is made that there is no user input, i.e., 
in the event that the user has made no operations, step 

15 S15003 through step S15005 are skipped, and the flow 
proceeds to step S15006. 

On the other hand, in step S15002, in the event that 
determination is made that there has been user input, i.e., 
in the event that the user has viewed the input image 

20 displayed on the image display unit 15015 and operated the 
user I/F 15016, thereby making user input indicating some 
sort of instruction or information, the flow proceeds to 
step S15003, where the user I/F 15016 determines whether or 
not the user input is user instructions instructing ending 

25 of the processing of the signal processing device 4. 
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In the event that determination is made in step S15003 
that the user input is ending instructions, the signal 
processing device 4 ends the processing. 

Also, in the event that determination is made in step 
5 S15003 that the user input is not ending instructions, the 
flow proceeds to step S15004, where the user I/F 15016 
determines whether or not the user input is constraint 
condition information. In the event that determination is 
made in step S15004 that the user input is not constraint 

10 condition information, step S15005 is skipped, and the flow 
proceeds to step S15006. 

Also, in the event that determination is made in step 
S15004 that the user input is constraint condition 
information, the flow proceeds to step S15005, where the 

15 user I/F 15016 supplies the constraint condition information 
to the equation generating unit 15022, and the flow proceeds 
to step S15006. 

In step S15006, the processing region setting unit 
15011 sets the processing region based on the input image, 

20 and supplies the processing region information identifying 
the processing region to the continuity setting unit 15012, 
the model generating unit 15021 and the equation generating 
unit 15022 of the actual world estimating unit 15013, and 
image generating unit 15014, and the flow proceeds to step 

25 S15007. Now, the setting of the processing region in step 
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S15006 can be arranged by the user inputting the processing 
region instruction information by operating the user I/F 
15016, and can be performed based on this processing region 
instruction information, or can be performed with no 
5 processing region instruction information from the user. 

In step S15007, the continuity setting unit 15012 
recognizes the processing region in the input image, from 
the processing region information supplied, from the 
processing region setting unit 15011. Further, the 

10 continuity setting unit 15012 sets the continuity of the 
signal of the actual world 1, of which one portion of 
continuity has been lost of the image data of the processing 
region, and the continuity information showing this 
continuity is supplied to the model generating unit 15021 of 

15 the actual world estimating unit 15013, and the flow 

proceeds to step S15008. Here the continuity setting unit 
15012 sets the movement amount v, expressing that the 
movement is in the horizontal direction at a constant speed 
of v pixels per shutter time (exposure time) , as the 

20 continuity information, and supplies this to the model 

generating unit 15021 of the actual world estimating unit 
15013. Here, the continuity setting unit 15012 sets the 
movement amount showing only the movement size as continuity 
information, under the premise that the object is moving in 

2 5 a horizontal direction, but a movement vector showing the 
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size and direction of the movement of the object can be set 
as continuity information as well. The setting of 
continuity in step S15007 can be arranged by the user 
inputting the continuity instruction information by 
5 operating the user I/F 15016, and can be performed based on 
this continuity instruction information, or can be performed 
with no continuity instruction information from the user. 

The actual world estimating unit 15013 performs actual 
world estimating processing in step S15008. In other words, 

10 with the actual world estimating unit 15013, the model 

generating unit 15021 generates an equation (model equation) 
as the model (relation model) wherein the relation of the 
pixel values of each of the pixels of the input image which 
has movement blurring and the signal of the actual world 1 

15 is modeled, based on the movement amount v supplied from the 
continuity setting unit 15012 in step S15007 and the 
processing region information supplied from the processing 
region setting unit 15011 in step S15006, and the generated 
model is supplied to the equation generating unit 15022. 

2 0 The equation generating unit 15022 generates a 

constraint condition expression based on the constraint 
condition information supplied from the user I/F 15016 in 
step S15005, and further, generates an equation for finding 
the approximation function of the model (approximation 

2 5 model) wherein the signal of the actual world 1 is modeled, 
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from the constraint condition expression and the model 
equation supplied from the model generating unit 15021 to 
the equation generating unit 15022, and substitutes the 
pixel values of each of the pixels of the input image into 
5 this equation, and supplies this to the actual world 
waveform estimating unit 15023. 

The actual world waveform estimating unit 15023 
estimates the waveform of the signal of the actual world 1, 
that is to say, finds the approximation function as a model 

10 which models the signal of the actual world 1, by 

calculating the equation supplied from the equation 
generating unit 15022, and supplies this approximation 
function to the image generating unit 15014 as the 
estimation result of the signal waveform of the actual world 

15 1. 

The actual world estimating processing of step S15008 
will be described in detail later with reference to Fig. 130. 

After the processing in step S15008, the flow proceeds 
to step S15009, and the image generating unit 15014 

20 generates a signal more closely approximating the signal 
from the actual world 1, from the approximation function 
supplied from the actual world waveform estimating unit 
15023, that is to say, generates an image with no movement 
blurring, and based on the processing region information 

25 supplied from the processing region setting unit 15011, 
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replaces the portion of the processing region of the input 
image with an image which has no movement blurring generated, 
generates this as an output image, and supplies this to the 
image display unit 15015, and the flow proceeds to step 
5 S15010. 

In step S15010, the image display unit 15015 displays 
the output image supplied from the image generating unit 
15014 instead of the input image displayed in step S15001, 
or along with the input image, and the flow proceeds to step 
10 S15011. 

In step S15011, as with the case of step S15002, the 
user I/F 12006 determines whether or not there has been some 
sort of user input by the user operating the user I/F 15016, 
and in the event that determination is made that there has 

15 been no user input, i.e., in the event that the user has 
made no operations, the flow returns to step S15011, and 
awaits some sort of user input. 

Also, in the event that determination is made in step 
S15011 that there has been user input, i.e., in the event 

2 0 that the user has viewed the input image or output image 

displayed on the image display unit 15015 and operated the 
user I/F 15016, thereby making user input representing some 
sort of instructions or information, the flow proceeds to 
step S15012, where the user I/F 15016 determines whether or 

2 5 not the user input is ending instructions instructing ending 



- 357 - 

S04P0850 

of the processing of the signal processing device 4. 

In the event that determination is made in step S15012 
that the user input is ending instructions, the signal 
processing device 4 ends processing. 
5 On the other hand, in the event that determination is 

made in step S15012 that the user input is not ending 
instructions, the flow proceeds to step S15013, where the 
user I/F 15016 determines whether or not the user input is 
constraint condition information. In the event that 

10 determination is made in step S15013 that the user input is 
not constraint condition information, the flow returns to 
step S15011, and the same processing is repeated thereafter. 

Also, in step S15013, in the event that determination 
is made that the user input is constraint condition 

15 information, the flow returns to step S15005, and as 

described above, the user I/F 15016 supplies the constraint 
condition information to the equation generating unit 15022. 
The flow then proceeds from step S15005 to step S15006, and 
hereafter the same processing is repeated. 

2 0 Now, in the event that the signal processing device 4 

repeats the processing of step S15005 through step S15013, 
the processing of the above-described step S15006 and S15007 
are the same as that set by the first processing S15006 and 
S15007, and the same process as the first process may be 

2 5 repeated, or may be skipped. 
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Thus, with the signal processing device 4 in Fig. 113, 
by finding an approximation function as a model wherein the 
signal of the actual world 1 is modeled, an output image of 
high image quality with no movement blurring can be 
5 generated. 

Also, by operating the user I/F 15016, the user can set 
constraint conditions which constrain the relation between 
each of the pixels in the original image having no movement 
blurring, and in the event that a new constraint condition 

10 is input, an image without movement blurring can be 

requested again, and therefore an output image of high image 
quality according to the preference of the user can be 
easily acquired. 

With the signal processing device 4 in Fig. 113, an 

15 output image without movement blurring is generated by 

finding an approximation function which approximates the 
light signal of the actual world 1, but this output image 
can be viewed as having the movement blurring removed from 
the input image. Accordingly, it can be said that 

2 0 processing is performed for movement blurring removal with 
the signal processing device 4 in Fig. 113. 

The processing for movement blurring removal with the 
signal processing device 4 in Fig. 113 will be described in 
detail below. First, by modeling (formulating) the 

25 mechanism for movement blurring generation, a relation model 
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is generated which models the relation between the pixel 
values of each of the pixels of the input image with 
movement blurring and the signal of the actual world 1. The 
light signal of the actual world 1 has no movement blurring, 
5 and therefore the relation model is a modeling of the 

relation between the pixel values of each of the pixels of 
the movement blurring image and the pixel values of each of 
the pixels of the image with no movement blurring. 

Fig. 115 is a diagram describing the light signal of 

10 the actual world 1. 

The light signal of the actual world 1 is to be 
regarded as a waveform F(x,y) on an X - Y plane (a plane 
formed of the X direction which is one direction of a 
spatial direction, and a Y direction which is perpendicular 

15 to the X direction) as illustrated in Fig. 115. The signal 
input into the signal processing device 4 of Fig. 113 is 
moving in the horizontal direction at the movement amount of 
v pixels as described above, and therefore is the same value 
as the waveform F(x,y) which is moving at the movement 

20 amount of" v pixels per shutter time in the same direction as 
the spatial direction X. Hereafter, the waveform F(x,y) 
will be called the light signal function F(x,y). 

The actual world estimating unit 15013 of the signal 
processing device 4 in Fig. 113 estimates a light signal 

25 function F(x,y) of the actual world 1 by finding an 
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approximation function f (x,y) which approximates the light 
signal function F(x,y) of the actual world 1, based on the 
input image from the sensor 2 . 

The input image which is the output of the sensor 2 is 
5 the image data configured with multiple pixels wherein the 
above-described light signal of the actual world 1 (the 
light signal function F(x,y)) is projected as pixel values 
from the detecting elements (pixels) of the sensor which has 
an integration effect. 

10 The integration effect in the case that the sensor 2 is 

a CCD will be described with reference to Fig. 116. 

As shown in Fig. 116, multiple detecting elements 
(pixels) 15031 are disposed on the plane of the sensor 2. 
In the example shown in Fig. 116, the direction 

15 parallel to 1 predetermined side of the detecting element 
15031 is the X direction which is one direction in the 
spatial direction, and the direction perpendicular to the X 
direction is the Y direction, which is the other direction 
in the spatial directions. The direction perpendicular to 

20 the X-Y plane is the t direction, which is the time 
direction. 

Also, with the example in Fig. 116, the spatial shape 
of the respective detecting elements 15031 (corresponding to 
the pixels of the image data output by the sensor 2) of the 
25 sensor 2 is taken as a square of which one side is 1 in 



- 361 - 



S04P0850 



length. The shutter time (exposure time) of the sensor 2 is 
taken as 1 . 

Further, with the example in Fig. 116, the center of 
one detecting element 15031 of the sensor 2 is taken as the 
5 origin (position x = 0 in the X direction, and position y = 
0 in the Y direction) in the spatial direction (X direction 
and Y direction) , and the intermediate point - in- time of the 
exposure time is taken as the origin (position t = 0 in the 
t direction) in the temporal direction (t direction) . 

10 In this case, the detecting element 15031 of which the 

center is in the origin (x = 0, y = 0) in the spatial 
direction subjects the light signal function F (x, y, t) to 
integration with a range between -0.5 and 0.5 in the X 
direction, range between -0.5 and 0.5 in the Y direction, 

15 and range between -0.5 and 0.5 in the t direction, and 
outputs the integral value thereof as a pixel value P. 

That is to say, the pixel value P output from the 
detecting element 15031 of which the center is in the origin 
in the spatial direction is represented with the following 

20 Expression (137) . 



. . . (137) 

The other detecting elements 15031 also output the 
pixel value P shown in Expression (13 7) by taking the center 
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of the subject detecting element 15031 as the origin in the 
spatial direction in the same way. 

Thus, the input image from the sensor 2 is a collection 
of data wherein each of the detecting elements (pixels) 
5 15031 has a constant pixel value due to the integration 

effect of the sensor 2. Also, the integration effect of the 
sensor 2 comprises the integration of the spatial direction 
(X-Y direction) of the light signal of the actual world 1 
(special mixture) and the integration in the time direction 

10 (t direction) (time mixture) . 

Accordingly, even in the cases wherein movement 
blurring of an object is occurring in an input image, 
strictly speaking the pixel value of each of the pixels of 
the input image is the value generated by both the spatial 

15 integration effect and the time integration effect (spatial 
mixture and time mixture have occurred) , but with the 
movement blurring due to the movement of an object in the 
horizontal direction, the effect of the time integration 
effect is great, and therefore with this embodiment, in 

2 0 order to model the generation of movement blurring more 

easily, let us say that the approximation function f (x,y) is 
modeled as having generated only the time integration effect 
by the object moving v pixels per shutter time. 

In other words, the approximation function f (x,y) 

25 approximating the actual world 1 has no spatial integration 
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effect, as illustrated in Fig. 117, that is to say, is a 
function having a constant value within the various pixels 
of the sensor, and let us say that pixel values of the 
various pixels of the input image with movement blurring 
5 occurring are time- integrated values by the approximation 
function f (x,y) in Fig. 117 moving in the horizontal 
direction at a constant speed at the movement amount v. 

Fig. 118 is a diagram illustrating the relation of the 
input image which is input in the signal processing device 4 

10 and the processing region which is set in the processing 
region setting unit 15011. 

The image 15 041 (input image) which is input from the 
sensor 2 (Fig. 1) into the signal processing device 4 is an 
image having a predetermined number of pixels in the spatial 

15 direction of the X direction and the spatial direction of 
the Y direction. 

A region 15042 which is a portion or all of the image 
15041 is set as the processing region by the process region 
setting unit 15011. The information illustrating the region 

20 15042 within the image 15041 is supplied as processing 

region information from the processing region setting unit 
15011 to the continuity setting unit 15012, the actual world 
estimating unit 15013, and the image generating unit 15014. 
Hereafter the region 15042 will be called the processing 

25 region 15042 . 
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Fig. 119 is a diagram illustrating the processing 
region 15042 . 

The processing region 15042 is a region having a 
processing region width of 1 = 10 pixels and line number of 
5 k = 4, for example, as illustrated in Fig. 119. In other 

words, in Fig. 119, the processing region 15042 is comprised 
of 10 x 4 = 40 pixels. 

Fig. 12 0 is a diagram illustrating the approximation 
function f (x,y) in Fig. 117 at the timing of a predetermined 

10 instant regarding one line wherein the value of y is the 

predetermined value y c (y = y c ) / within the processing region 
15042. In other words, Fig. 120 is a diagram illustrating 
the approximation function f (x, y c ) at the timing of a 
predetermined instant. This approximation function f (x, y c ) 

15 is an approximation function f (x, y) illustrating the 
relation pixel level and the position of the special 
direction X when the time direction t and the spatial 
direction Y is at a fixed value, and this will simply be 
called an approximation function f (x) . 

2 0 The approximation function in Fig. 12 0 can be expressed 

as the following Expression (138) . 
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. . . (138) 

Expression (138) is an approximation function f (x) at 
the timing of an instant, and actually, the approximation 
function f (x) has a movement amount v, and therefore after a 
1/v time in Fig. 120, becomes such as that illustrated in 
Fig. 121. The approximation function in Fig. 12 0 can be 
expressed as the following Expression (139) . 
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. . . (139) 

Further, after a 1/v time in Fig. 121, the function 
becomes such as that illustrated in Fig. 122, and the 
5 approximation function f (x) at this time is represented with 
the following Expression (140) . 
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. . . (140) 

Here the movement amount v of the object can be 
considered as a simplified form as illustrated in Fig. 123, 
by the movement of the approximation function f (x) 
5 illustrated in Fig. 120 through Fig. 122. In other words, 
the object moves instantly to the center of the pixel next 
over in the x direction every 1/v time, and stays there for 
the period l/v time. Accordingly, during the 1/v time, the 
pixel value of the same value as when the object is not 
10 moving as accumulated in the pixels of the sensor. Also, 

during the next 1/v time, the pixel values that are the same 
as when the object is not moving, is accumulated as to the 
pixel that was advanced only one pixel worth in the x 
direction. 

15 In other words, in the event that the object (the light 

signal of the actual world 1) is moving at the movement 
amount v, each pixel can be considered to have 1/v times the 
pixel value as the same pixel value when the object is not 
moving at the movement amount v, accumulated in order, for 

2 0 every 1/v time. 

As described above, even if we think of the object as 
moving only one pixel worth in the x direction instantly 
every 1/v time, the object can be seen as moving at the same 
speed in the x direction with the movement amount of v 

25 pixels for each unit of time, when viewed at a macro level, 
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as illustrated by a straight line in Fig. 123. 

The accumulation by the detecting elements of the 
sensor (time integration) of the object having the movement 
amount v will be further described with reference to Fig. 
5 124 through Fig. 126. 

Fig. 124 is a diagram showing the approximation 
function f (x) of y = y c at a predetermined precise timing, 
illustrated in Fig. 120, with a X - t plane. 

In Fig. 124, the width of the X direction of each pixel 
10 15031 is 1, as described in Fig. 116, and the height in the 
t-direction is the shutter time (=1) . Also, the area (the 
width in the X direction x the height in the t-direction) of 
each pixel 15031 are each the pixel values Q 0 through Q 9 
accumulated (expressed by the total amount of electric 
15 charge) in each pixel 15031. These pixel values Q 0 through 
Q 9 are the pixel values of the image with no movement 
blurring . 

On the other hand, Fig. 125 is a diagram illustrating 
the pixel values, which are time- integrated and accumulated, 

2 0 when the object is moving at the movement amount v so as to 
correspond to the pixel values Q 0 through Q 9 in Fig. 124, as 
P 0 through P 9 . In other words this is a diagram of an X-t 
plane of the pixel values P 0 through P 9 of the input image, 
regarding one line of the y = y c within the processing 

25 region 15042 . 
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For example, based on the movement of the object 
described in the above -described Fig. 12 0 through Fig. 123 
with v = 4 (pixels) , when the pixel values P 0 through P 9 of 
Fig. 125 are expressed using the pixel values Q 0 through Q 9 
5 in Fig. 124, the result is such as that in Fig. 126. 

In other words, for the first 1/v time of the shutter 
time, the pixel values (the electric charge corresponding 
thereto) equivalent to l/v of the pixel value Q h (h = 0 
through 9) with no movement blurring are accumulated into 

10 each of the pixels where x = h. Then, for the next l/v time, 
the pixels move over one pixel worth of the movement 
direction side in the X direction (the pixel on the right 
side in the diagram) , and the pixel values equivalent to l/v 
of the same pixel value Q h (h = 0 through 8) are accumulated 

15 into each of the pixels where x = h. Hereafter, similarly, 
for each l/v time, as the pixels move over one pixel worth 
of the movement direction side in the X direction, the pixel 
values equivalent to l/v of the pixel value Q h (h = 0 
through 8) are accumulated. Hereafter, as long as the 

2 0 variable h which is a reference numeral has no special 

restriction, the variable h will express an integer where h 
= 0 through 9 . 

In other words, with the pixel wherein X = 0 is the 
center thereof (hereafter, this will be called a pixel with 

25 X = 0) , l/v of the pixel value of the pixel value Q 0 (Qo/v) 
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is accumulated. With the pixel wherein X = 1 is the center 
to the immediate right of the pixel wherein X = 0 (hereafter 
this will be called the X = 1 pixel) , the pixel value (Qi/v) 
which is 1/v of the pixel value Q x and the pixel value (Qi/v) 
which is 1/v of the pixel value Q 0 are accumulated. With 
the pixel wherein X = 2 is the center to the immediate right 
of the pixel wherein X = 1 (hereafter this will be called 
the X = 2 pixel) , the pixel value (Q 0 /v) which is 1/v of the 
pixel value Q 0 , the pixel value (Qi/v) which is 1/v of the 
pixel value Q lf and the pixel value (Q 2 /v) which is l/v of 
the pixel value Q 2 are accumulated. Thereafter, the pixel 
values which are l/v of the pixel values Q h are similarly 
moved over each l/v time for each pixel, and accumulated. 

The following Expression (141) is acquired when 
expressing the relation illustrated in Figs. 125 and 126 
with an expression. 

(Qo+Qt +02+03)/^ = P 3 
(0^02+03+04)^ = P 4 
(Q 2 +Q 3 +Q 4 +Q 5 )/v = P 5 
(Q 3 +Q 4 +Q 5 +Q 6 )/v = P 6 
(Q 4 +Q 5 +Q 6 +Q 7 )/v = P 7 
(Q 5 +Q 6 +Q 7 +Q 8 )/v = P 8 
(Q 6 +Q 7 +Q 8 +Q 9 )/v = P 9 

. . . (141) 

Here, with the pixels wherein x = 0, 1, or 2 , there is 



- 371 -. 

S04P0850 

a blank (unknown) region of the accumulated pixel values. 
Therefore, each of the P 0/ Pi, and P 2 has not acquired an 
accumulated pixel value within the entire shutter time, and 
so cannot build an expression. 
5 Expression (141) is a simultaneous equation wherein the 

mechanism of movement blurring generation is modeled, with 
the pixel values of each of the pixels of the movement 
blurring image being considered to be the accumulated 
(integrated) value wherein the pixel values of each of the 

10 pixels of an image with no movement blurring are accumulated 
(integrated) , while moving at the movement amount v. In 
Expression (141) , the pixel value P h when the object is 
moving at the movement amount v is the pixel value already 
known of the input image to be input into the signal 

15 processing device 4, and the pixel values Q h when the object 
has no movement expresses the pixel value to be found with 
no blurring. 

Thus, the pixel values P h when the object is moving at 
the movement amount v can be said to be the pixel value 

2 0 wherein the pixel values Q h when the object has no movement 
are mixed for l/v time each (time mixture) . Hereafter, the 
pixel value P h when the object is moving at the movement 
amount v is called the mixed pixel value P h , and the pixel 
values Q h when the object has no movement is simply called 

25 the pixel values Q h . 
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The approximation function f (x) as an approximation 
model wherein the signal of the actual world 1 is modeled is 
configured with a value of Q h , as expressed in Expression 
(139) with the above-described example, and so Expression 
5 (141) is a model equation as a relation model wherein 

relation of the pixel values of each of the pixels of the 
movement blurring image as the input image and the signal of 
the actual world 1 is modeled. Also, finding the pixel 
value Qh with no movement blurring is the same as finding 
10 the approximation function f (x) wherein the actual world 

estimating unit 15013 has modeled the signal of the actual 
world 1 . 

The above-described simultaneous equation (141) can be 
built by setting the processing region width 1 of the 

15 processing region 15 042 and the movement amount v. However, 
the simultaneous equation (141) only has 7 expressions, 
compared to the 10 variables of the pixel values Q h . 
Accordingly, it is difficult to find the pixel values Qh 
with only Expression (141) . 

20 Therefore, two methods will be described for having at 

least the same number of expressions and number of variables. 

The first method is a method of hypothesizing that the 
edge portions of the processing region is "flat". That is 
to say, as shown in Fig. 127, this is a method of 

25 hypothesizing that the accumulated pixel value as to the 
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blank (unknown) region 15051 of the pixel value is the same 
as the pixel value (Q 0 /v) of l/v of the pixel value Q 0/ which 
is the edge portion of the first l/v time. 

In this case, the following Expression (142) can be 
newly acquired. 



From Expression (141) and Expression (142) , the number 
of expressions is 10 and the number of variables is 10, and 
so the pixel value Q h can be found. 

With the processing which is described with reference 
to the flowchart in Fig. 114, in the event that there is no 
specification (setting) for constraint conditions by the 
user, that is to say, in the event that constraint condition 
information of the equation generating unit 15022 is not 
supplied form the user I/F 15016, for example the first 
method may be used. However, with the first method, the 
edge portions of the processing region are hypothesized as 
being "flat", and so the image configured from the pixel 
value Q h to be found with the first method can have noise 
generated in some cases . 

Thus, in the case that constraint condition information 
is supplied from the user I/F 15016 to the equation 



(G 0 +Q 0 +Go+Qo) /V 
(Q 0 +Qb+Q 0 +Qi)/v 

(Qo+Qo+Qi+Q2)/v 




(142) 
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generating unit 15022, a second method can be applied which 
is different from the first method. The second method is a 
method for acquiring a number of expressions greater than 
the number of variables by combining Expression (141) with 
an Expression (hereafter referred to as a constraint 
condition expression) , which is built that hypothesizes 
(constrains) the relation of the pixel values of the 
adjacent various pixels in the image with no blurring. 

In general, an image often has a spatial correlation, 
and therefore adjacent pixels often do not differ greatly. 
Thus, for example, a condition (hereafter referred to as a 
constraint condition) can be hypothesized to say that the 
difference is 0 (hereafter this will be called "adjacent 
pixel difference = 0") for the pixel value Qh between each 
of the pixels not generating movement blurring. This 
constraint condition can be expressed with the following 
Expression (143) . 

G h ^-Q h = 0 h-1,- • (1-1) 

... (143) 

Also, for example, a condition stating that "the pixel 
values are changing smoothly" can be hypothesized. This 
constraint condition can be expressed with the following 
Expression (144) . 

-Qh-! +2Q h -Q h+1 =0 h = 1 , • • • , (I -2) 

. . . (144) 
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The user viewing an image displayed on the image 
display unit 15015 reflecting a blurred object guesses the 
original image, that is to say, an image without movement 
blurring which is an image of the actual world 1 
5 corresponding to the input image, and can set the relation 
equation (constraint conditions) between the pixels which 
expresses the guessed state, into the equation generating 
unit 15022, by operating the user I/F 15016. 

With the signal processing device 4 in Fig. 113, the 

10 constraint condition of Expression (143) is said to be set 
by the user (the constraint condition information is 
supplied 'from the user I/F 15016 which instructs Expression 
(143)). In this event, regarding the line illustrated in 
Fig. 124, a constraint condition expression can be acquired 

15 with is expressed with the following Expression (145) . 

Q 0 -Q 1 = 0 
Qi-Q 2 = 0 
Q 2 -Q 3 = 0 
Q3-Q4 = 0 
Q4-Q5 = 0 
Q 5 -Q 6 = 0 
Q 6 -Q 7 = 0 
Q 7 -Q 8 = 0 
Q 8 -G 9 = 0 

. . . (145) 
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Expression (14 5) has 9 expressions, and when added to 
Expression (141) , the number of expressions is 16, and the 
number of variables Q h is 10, and so for example, by using 
the least -square method, a variable Q h wherein the sum of 
squares of error generated in the various expressions is the 
smallest, can be calculated. In other words, the pixel 
value Qh with the movement blurring removed (with no 
movement blurring) may be estimated. 

Specifically, if the error generated in each of the 
expressions in Expression (141) is e m i (i = 3 through 9) , and 
the error generated in each of the expressions in Expression 
(14 5) is e bj (b = 0 through 8) , Expression (141) and 
Expression (145) can be expressed by Expression (146) and 
Expression (147) , respectively. 

(Q 0 +Q 1 +Q 2 +Q3)'''v = P 3 +e m3 
(Q 1 +Q 2 +Q 3 +Q 4 )/v = P 4 +e m4 
(Q 2 +Q 3 +Q 4 +Q 5 )/v = P 5 +e m5 
(Q 3 +Q 4 +Q 5 +Q 6 )/v = P 6 +e m6 

(Q 4 +Q 5 +Q 6 +Q 7 )/v = P 7 +e m7 
(Q 5 +Q 6 +Q 7 +Q 8 )/v = P 8 +e m8 
(Q 6 +Q 7 +Q 8 +Q 9 )/v = P 9 +e m9 

. . . (146) 
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From Expression (14 6) and Expression (147) , the 
following Expression (148) holds. 
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. . . (148) 

Hereafter, as long as the variables i and j have no 
special restrictions, the variable i will express an integer 
where i = 3 through 9 and j will express an integer where j 
= 0 through 8 . 

Expression (148) is replaced with Expression (149) , 
using a matrix A, and column vectors x, y, and e. 

Ax = y+e 

. . . (149) 

In this case, the total sum E of the squared errors of 
the error e m i of Expression (14 6) and the error e b j of 
Expression (147) can be expressed with the following 
expression . 

E=|e| 2 =Ie mi 2 +Ie bj 2 
= (Ax-y) T (Ax-y) 
= x T A T Ax-2x T A T y+y T y 

. . . (150) 

In order to find the Q h for which the total sum E of the 
squared error is the smallest, the fact that the total sum E 
of the squared error of the equation (15 0) can have 0 as the 
value partially differentiated with the column vector x with 
Q h as a component leads into the following expression. 

ff = 2(A T Ax-A T y) =0 
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. . . (151) 

From Expression (151) , the column vector x (a column 
vector x with the pixel value of Q h as a component) to be 
found can be expressed with Expression (152) . 

x= (A T A)" 1 A T y 

. . . (152) 

Now, with Expression (152) , the superscript T means 
transposing, and the superscript -1 means an inverse matrix. 

By substituting the column vector y, of which the mixed 
image value Pi is a component, into Expression (152) and 
computing this, the pixel value Q h with the movement 
blurring removed (with no movement blurring) can be 
estimated . 

Fig. 128 is a diagram illustrating again the processing 
region 15042 . 

A column vector x (= pixel value Q h ) which can be found 
from Expression (152) is a column vector x (= pixel value 
Q h ) of one line of the y = y c of the processing region 15042 
illustrated in Fig. 128, and therefore in order to estimate 
a pixel value with no movement blurring for all pixels in 
the processing region 15042, the computation of Expression 
(152) must be repeated the same number of times k (=4) as 
the line number. 

In other words, as with the y = y c line in the above 
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description, if the one line which performs processing to 
find the pixel value Q h with no movement blurring is called 
the line of interest, the actual world estimating unit 15013 
sets each of all of the lines, in order, as the line of 
5 interest, and by performing processing to estimate the pixel 
values Q 0 through Q 9 , which is the approximation function 
f (x) of the line of interest, an image with no movement 
blurring can be estimated regarding the processing region 
15042 of the input image. 

10 Fig. 129 is a configuration example of the inside of 

the actual world estimating unit 15013 in Fig. 113. 

The actual world estimating unit 15013 comprises a 
model generating unit 15021, an equation generating unit 
15022, and an actual world waveform estimating unit 15023. 

15 Further, the equation generating unit 15022 comprises a 
constraint condition setting unit 15061 and a normal 
equation generating unit 15062. 

The movement amount v as the continuity information is 
supplied to the model generating unit 15021 from the 

20 continuity setting unit 15012 as continuity information. 
The movement amount v in this example expresses that the 
object is moving in the horizontal direction only v pixels 
per shutter time, within the processing region of the input 
image which is set by the processing region setting unit 

25 15011, and for example, v = 4, as with the above-described 
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example. Also, the movement amount v may be a value which 
differs for each line, and in the event that the movement 
amount v differs for each line, k number, equal to the 
number of lines, of the movement amount v is supplied as 
5 continuity information from the continuity setting unit 
15012 to the model generating unit 15021. 

Also, processing region information which specifies the 
processing region within the input image is also supplied to 
the model generating unit 15021 from the processing region 

10 setting unit 15011. In the case of using a processing 

region with a rectangular-shaped region as the processing 
region, the processing region information can be expressed, 
for example, by coordinate values at opposite angles of a 
rectangular- shaped processing region as to an absolute 

15 coordinates system wherein a predetermined position (pixel) 
of the input image is the origin, or the center coordinate 
of the processing region and the width (horizontal 
direction) and the height (vertical direction) of the 
processing region, or the like. 

20 The model generating unit 15021 acquires, for example, 

a processing region width of 1, and k line numbers, as 
illustrated in Fig. 119, from the processing region 
information supplied from the processing region setting unit 
15011. In this example, let us say that the processing 

25 region 15042 is rectangular- shaped, and for example, has 1 = 
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10 and k = 4. In this case, the processing region width of 
each line (the number of pixels) are the same, but with the 
processing region 15042, each line can have a different 
processing region width. In the case that the processing 
5 region width 1 differs for each line, the model generating 
unit 15021 acquires a processing region width 1 of the same 
number (k) as the number of lines. 

Also, the model generating unit 15021 generates an 
equation (model equation) of a mixed pixel value Pi which is 

10 shown in the above -described Expression (141) , based on the 
movement amount v supplied from the continuity setting unit 
15012 and the processing region width 1 and k number of 
lines acquired from the processing region information, and 
supplies this to the normal equation generating unit 15062 . 

15 The constraint condition information specified by the 

user operating the user I/F 15016 is supplied to the 
constraint condition setting unit 15061 from the user I/F 
15016. For example, the user performs an operation with the 
user I/F 15016 to select (specify) the desired constraint 

20 condition from the constraint conditions which constrain the 
relation of the pixel values Qh of each of the pixels of an 
image with no movement blurring, such as the above-described 
Expression (143) or Expression (144) , which have been set in 
advance inside the constraint condition setting unit 15061, 

25 and the user I/F 15016 supplies the information describing 
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the selected constraint conditions as the constraint 
condition information to the constraint condition setting 
unit 15061. 

The constraint condition setting unit 15061 selects a 
5 constraint condition expression corresponding to the 

constraint condition information supplied from the user I/F 
15016 from the constraint condition expressions within the 
constraint condition setting unit 15061 itself. Further, 
the constraint condition setting unit 15061 acquires the 

10 processing region width 1 based on the processing region 
information of the input image supplied from the equation 
generating unit 15022 from the processing region setting 
unit 15011, and generates only the number, which corresponds 
to the processing region width 1, of the constraint 

15 condition expressions. For example, in the event that 

Expression (143) is used which has the condition "adjacent 
pixel difference = 0" as the constraint condition 
information, the constraint condition setting unit 15061 
generates the constraint condition expression expressed by 

2 0 the above -described Expression (14 5) , by acquiring the 

processing region width 1 based on the processing region 
information supplied from the processing region setting unit 
15011, and supplies this to the normal equation generating 
unit 15062. In the example of the processing region 15042 

25 illustrated in Fig. 119, the processing region 15042 is 
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rectangular- shaped, and the processing region width 1 is the 
same of each line, but in the case that the processing 
region width 1 differs for each line, the number of 
constraint condition expressions also differ for each line. 
5 The equation with k number of mixed pixel value Pi, 

which is the same as the number of lines, (Expression (141)) 
from the model generating unit 15021 and the constraint 
condition Expression (Expression (145) ) corresponding to the 
constraint condition selected by the user operating the user 

10 I/F 15016 from the constraint condition setting unit 15061 
are supplied to the normal equation generating unit 15062 . 

First, as illustrated in Fig. 128, the normal equation 
generating unit 15062 selects a predetermined line from the 
processing region 15042 as a line of interest. Then the 

15 normal equation generating unit 15062 generates a normal 
equation corresponding to the selected line of interest, 
which is expressed by Expression (148) (Expression (149)), 
from a mixed pixel value Pi equation expressed by Expression 
(141) which is supplied from the model generating unit 15021, 

20 and a constraint condition expression expressed by 

Expression (145) which is supplied from the constraint 
condition setting unit 15061. 

Further, the normal equation generating unit 15062 
computes the matrix (A T A) _1 A T , which is a coefficient of the 

25 column vector y of the right side of Expression (152) , and 
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substitutes the mixed pixel value Pi which is acquired based 
on the input image supplied from the sensor 2 (Fig. 1) into 
the column vector y. The matrix (A T A) -1 A T is computed, and 
Expression (152) into which the mixed pixel value Pi is 
5 substituted is further supplied from the normal equation 
generating unit 15062 to the actual world waveform 
estimating unit 15023. 

When the normal equation generating unit 15062 finishes 
the above-described generation of the normal equation, the 
10 computation of the matrix (A T A) _1 A T , and the processing such 
as the substitution of the mixed pixel value Pi, a line not 
yet selected is selected as the next line of interest, and 
similar processing is performed as to the selected next line 
of interest. 

15 The actual world waveform estimating unit 15023, by 

computing Expression (152) which is supplied from the normal 
equation generating unit 15062, finds the column vector x, 
that is to say, the pixel value Q h with no movement blurring 
(approximation function) , and supplies this to the image 

20 generating unit 15014 (Fig. 113) . 

The image generating unit 15014 replaces a portion of 
the processing region 15042 of the input image with the 
image of the pixel value Q h with no movement blurring, which 
is supplied from the actual world waveform estimating unit 

25 15023, based on the processing region information supplied 
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from the processing region setting unit 15011, and generates 
this as an output image, and supplies this to the image 
display unit 15015. 

Next, the actual world estimating processing of the 
5 actual world estimating unit 15013 in step S15008 in Fig. 
114 will be described inn detail with reference to the 
flowchart in Fig. 13 0. 

First, in step S15031, the model generating unit 15021 
acquires the processing region width 1 and the number of 
10 lines k, based on the processing region information supplied 
from the processing region setting unit 15011, and the flow 
proceeds to step S15032 . 

In step S15032, the model generating unit 15021 
acquires the movement amount v, which is equivalent to the 
15 number of lines k in the processing region 15042, from the 

continuity setting unit 15012, and the flow proceeds to step 
S15033 . 

In step S15033, the model generating unit 15021 
generates an equation for the mixed pixel value P± for each 
20 line, and the flow proceeds to step S15034. In other words, 
the model generating unit 15021 generates k number of model 
equations (expressions (141) ) . 

In step S15034, the constraint condition setting unit 
15061 selects the constraint condition expression 
25 corresponding to the constraint condition information 
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supplied from the user I/F 15016, from the constraint 
condition expressions existing within the constraint 
condition setting unit itself. Also, the constraint 
condition setting unit 15061 acquires the processing region 
5 width 1 from the processing region information of the input 
image which is supplied from the equation generating unit 
15022 from the processing region setting unit 15011, and 
generates an constraint condition Expression (Expression 
(145) ) corresponding to the constraint condition information 

10 from the users interface 15016, which comprises a number of 
equations corresponding to the processing region width 1, 
and the flow proceeds to step S15035. 

In step S15035, the normal equation generating unit 
15062 selects one predetermined line as the line of interest 

15 from the processing region 15042, and the flow proceeds to 
step S15036. 

In step S15036, the normal equation generating unit 
15062 generates a normal equation expressed by Expression 
(148) (Expression (149)) from the equation (model equation) 

2 0 of the mixed pixel value Pi expressed by Expression (141) 

which is supplied from the model generating unit 15021, and 
the equation (constraint condition expression) of the 
constraint condition expressed by Expression (145) which is 
supplied from the constraint condition setting unit 15061, 

25 and the flow proceeds to step S15037. 
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In step S15037, the normal equation generating unit 
15062 computes the matrix (A T A) _1 A T , which is a coefficient 
of the column vector y of the right side of Expression (152) , 
and the flow proceeds to step S15038. 
5 In step S15038, the normal equation generating unit 

15062 substitutes the mixed pixel value P± which is acquired 
based on the input image supplied from the sensor 2 (Fig. 1) 
into the column vector y, and the flow proceeds to step 
S15039 . 

10 In step S15039, the actual world waveform estimating 

unit 15023 computes a normal equation. That is to say, the 
actual world waveform estimating unit 15023 find a pixel 
value Q h with no movement blurring by computing Expression 
(152) which is supplied from the normal equation generating 

15 unit 15062, and the flow proceeds to step S15040. 

In step S15040, determination is made as to whether or 
not processing is performed to find a pixel value Q h with no 
movement blurring as to all lines of the processing region 
15042. In the event that it is determined that processing 

2 0 is not performed to find a pixel value Q h with no movement 

blurring as to all lines of the processing region 15042, the 
process is returned to step S15035, and the processing in 
steps S15035 through S15040 is repeated. That is to say, a 
line not yet selected of the processing region 15042 is 

25 selected as the next line of interest, and processing is 
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performed to find a pixel value Q h with no movement blurring. 

On the other hand, in the event that it is determined 
that processing is performed to find a pixel value Q h with 
no movement blurring as to all lines of the processing 
5 region 15042, the actual world estimating processing ends 
and returns . 

Thus, by generating a normal equation (Expression 
(148) ) as to one line which is the line of interest of the 
processing region 15042, and repeating the processing to 
10 find a pixel value Q h with no movement blurring k number of 

times, which is the number of lines in the processing region 
15042, the pixel value Q h with no movement blurring can be 
found across the entire region of the processing region 
15042 . 

15 In the event that the movement amount v and the 

processing region width 1 are each the same for each line of 
the processing region 15042, the matrix A in Expression 
(148) is the same for every line. Accordingly, the matrix 
(A T A) _1 A T in Expression (148) also is the same for every line, 

20 and the processing to compute the matrix (A T A) _1 A T for each 
line can be omitted. 

Next, the actual world estimating processing of the 
actual world estimating unit 15013 in step S15008 in Fig. 
114 in the event that the movement amount v and the 

25 processing region width 1 are each the same for each line of 



- 390 - 

S04P0850 

the processing region 15042 will be described with reference 
to the flowchart of Fig. 131. However, the portions in Fig. 
131 which are similar to the flowchart in Fig. 130 will omit 
the description thereof as appropriate. 
5 In Fig. 131, the processing in step S15057 which 

corresponds to step S15035 in Fig. 130 is positioned not 
between step S15054 and S15055, but between steps S15056 and 
S15058. Further, in step S15060, if it is determined that 
processing is not performed to find a pixel value Qh with no 

10 movement blurring as to all lines of the processing region 
15042, the process is returned to step S15057. 

In other words, in the event that the movement amount v 
and the processing region width 1 of each line of the 
processing region 15042 are each the same, the normal 

15 equation generated in step S15055 is the same for each line, 
and therefore, after computing the matrix (A T A) X A T as to the 
first line of interest in step S15056, computing the matrix 
(A T A) _1 A T as to the remaining lines of interest (the 
processing in step S15056) is not necessary. Accordingly, 

20 the selection of the line of interest (step S15057) , the 

substitution of the mixed pixel value Pi (the processing in 
step S15058) , and the computation of the normal equation 
(the processing in step S15059) only need be executed in 
order, as to the lines not selected in the processing region 

25 15042. 
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Thus, with the embodiment illustrated in Fig. 113, a 
normal equation is generated from an equation of the mixed 
pixel value Pi wherein a pixel value generating movement 
blurring is modeled with a pixel value with no movement 
5 blurring (model equation) , and a constraint condition 

expression using the relation between adjacent pixels of a 
pixel value with no movement blurring, and by computing this 
normal equation, a pixel value with no movement blurring 
(approximation function) is found, and therefore an image 
10 with good image quality can be generated. 

Figs. 132 through 134 are diagrams to describe the 
processing results by the signal processing device 4 in Fig. 
113 . 

Fig. 132 illustrates an input image to be input in the 
15 signal processing device 4 in Fig. 113. With the input 

image in Fig. 132, the characters "TAL" are moving in the 
direction from left to right at the same speed, and movement 
blurring is occurring in the horizontal direction. 

Fig. 133 is an image using the above -described first 
2 0 method, that is to say, a expression hypothesizing that the 
edge portion of the processing region is flat is added to 
the model equation from the input image in Fig. 132 and a 
normal equation is generated, and the image in Fig. 133 is 
generated by computing the normal equation. As is clear by 
25 comparing Fig. 132 and Fig. 133, in the image in Fig. 133, 
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the movement blurring occurring in the input image in Fig. 
132 is reduced, and the characters "TAL" are relatively 
clearly displayed. 

On the other hand, Fig. 134 is an image using the 
5 above-described second method, that is to say, a normal 
equation is generated by adding a conditional expression 
constraining the relation of the adjacent pixel values from 
the input image in Fig. 132 to the model equation, and the 
image in Fig. 134 is generated by computing the normal 

10 equation. In the image in Fig. 134 also, the movement 

blurring occurring in the input image in Fig. 132 is reduced, 
and the characters "TAL" are clearly displayed. 

With the embodiments in Figs. 132 through 134, the 
image in Fig. 134 by the second method is an image with has 

15 less noise and no movement blurring as compared to the image 
in Fig. 133 by the first method. 

In the above example, the constraint condition 
information supplied from the user I/F 15016 to the 
constraint condition setting unit 15061 is said to be 

2 0 information expressing which constraint condition the user 
has selected (specified) from the several constraint 
conditions which are set in advance internally in the 
constraint condition setting unit 15061, but it can be such 
that the user can directly input the constraint condition 

25 expression by operating the user I/F 15016. 
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With the embodiment of the signal processing device 4 
illustrated in Fig. 113, an image with no movement blurring 
is generated such that the total sum E of the squared error 
expressed with Expression (150) becomes as small as possible, 
5 that is to say, generated such that the error occurring in 
the equation of the mixed pixel value Pi and the error 
occurring in the constraint condition expression becomes as 
small as possible. Corresponding thereto, for example, of 
the error e m i occurring in the equation of the mixed pixel 

10 value Pi of Expression (14 6) and the error e b j occurring in 

the constraint condition expression of Expression (147) , one 
error or the other can be made as small as possible, and 
thus the balance (weighting) of the model equation and the 
constraint condition expression can be adjusted and thus 

15 generate an image with no movement blurring. 

Fig. 135 illustrates such a configuration example of 
the embodiment of the signal processing device 4, which 
generates an image with no movement blurring by adjusting 
the weighting of the equation of the mixed pixel value Pi 

20 (the model equation) and the equation of the constraint 
conditions (constraint condition expression) . 

In other words, Fig. 135 illustrates a configuration 
example of another embodiment of the application example of 
the signal processing device 4 illustrated in Fig. 111. 

25 An image of one frame or one field, for example, as the 
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data 3 of the sensor 2 is input into the signal processing 
device 4 in Fig. 135. Here, let us say that an object 
having a predetermined shape which moves at a fixed speed of 
v pixels per shutter time (exposure time) in the horizontal 
5 direction (sideways direction) of the image is displayed in 
the input image. That is to say, the object is moving at 
the movement amount of v pixels in the horizontal direction, 
and therefore the light signal of the object and the light 
signal of the portions other than the object are mixed (time 

10 mixture) by the time- integration effect of the sensor 2, and 
thus, the input image becomes a blurred image in portions 
such as the edge portions of the object. The signal 
processing device 4 shown in Fig. 13 5 generates high-quality 
output images wherein movement blurring occurring due to 

15 such temporal mixing has been removed from the input image. 

In Fig. 135, the processing region setting unit 15081, 
the continuity setting unit 15082, the actual world 
estimating unit 15083, image generating unit 15084, image 
display unit 15085, and user I/F 15086, each correspond to 

20 the processing region setting unit 10001, the continuity 

setting unit 10002, the actual world estimating unit 10003, 
image generating unit 10004, image display unit 10005, and 
user I/F 10006 shown in Fig. Ill, and basically perform the 
same processing as the processing region setting unit 10001, 

25 the continuity setting unit 10002, the actual world 
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estimating unit 10003, image generating unit 10004, image 
display unit 10005, and user I/F 10006. Further, in Fig. 
135, the actual world estimating unit 15083 comprises a 
model generating unit 15091, an equation generating unit 
5 15092, and actual world waveform estimating unit 15093. The 
model generating unit 15091, equation generating unit 15092, 
and actual world waveform estimating unit 15093 correspond 
to each of the model generating unit 10011, equation 
generating unit 10012, and actual world waveform estimating 

10 unit 10013 in Fig. Ill, and basically perform the same 
functions as each of the model generating unit 10011, 
equation generating unit 10012, and actual world waveform 
estimating unit 10013. 

Note however, that in Fig. 135, the assisting 

15 information which the user I/F 15086 outputs by the user 
operating the user I/F 15086, is supplied to the equation 
generating unit 15092. 

In other words, in Fig. 135, by operating the user I/F 
15086, with the equation generating unit 15092, the user can 

20 set the constraint conditions which constrain the relation 
between each of the pixels of an image with no movement 
blurring, which is an image of the actual world 1 
corresponding to the input image, and when the user performs 
an operation to set the constraint conditions, the user I/F 

25 15086 supplies the constraint condition information 
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expressing the constraint conditions set by the operation to 
the equation generating unit 15092. 

Specifically, for example, a user who views an image 
reflecting a blurring image displayed on the image display 
5 unit 15085 guesses the original image, that is to say, an 
image without movement blurring which is an image of the 
actual world 1 corresponding to the input image. Also, for 
example, the user guesses predetermined regions where 
blurring is occurring as the edge portions, and guesses the 

10 relation of the pixel values between each of the pixels (the 
constraint conditions) , such as the difference being large 
between the pixel values (level) between the adjacent pixels 
at the edge portions. Further, the user can set the 
conditions suitable for the guessed relation of the pixels 

15 value between each of the pixels into the equation 

generating unit 15092, by operating the user I/F 15086. 

Also, in Fig. 135, the processing region information, 
which is supplied from the processing region setting unit 
15081 to the actual world estimating unit 15083, is supplied 

20 to the model generating unit 15091 and the equation 

generating unit 15092 of the actual world estimating unit 
15083. The model generating unit 15091 and the equation 
generating unit 15 0 92 recognize the processing region in the 
input image from the processing region information which is 

25 supplied from the processing region setting unit 15081. 
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The continuity setting unit 15082 sets a movement 
amount (movement vector) v in the input image as the 
continuity information that an object is moving in the 
horizontal direction of the image at the constant speed of v 
pixels per shutter time, and supplies this movement amount v 
as the continuity information to the model generating unit 
15091 of the actual world estimating unit 15083. 

The model generating unit 15091 generates an equation 
(hereafter referred to as model equation as appropriate) as 
a model (hereafter referred to as relation model as 
appropriate) , wherein the relation between the signal of the 
actual world 1 and the pixel values of each of the pixels of 
the movement blurring image as the input image which 
considers the movement amount v as the continuity 
information supplied from the continuity setting unit 15082, 
regarding the processing region recognized by the processing 
region information which is supplied from the processing 
region setting unit 15081, and supplies the generated model 
equation to the equation generating unit 15092. 

The equation generating unit 15092 generates a 
constraint condition Expression (hereafter referred to as 
constraint condition expression as appropriate) , based on 
the constraint condition information supplied from the user 
I/F 15086. Also, the equation generating unit 15092 sets 
the weighting into the constraint condition expression for 
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adjusting the balance of the model equation supplied from 
the model generating unit 15091, and substitutes the pixel 
values of each of the pixels of the input image comprising 
the processing region into the equation formed from the 
5 weighted constraint condition expression and the model 
equation, and thus, generates an equation to find the 
approximation function as a model (hereafter referred to as 
approximation model) wherein the signal of the actual world 
1 is modeled, and supplies this to the actual world waveform 

10 estimating unit 15093. 

The actual world waveform estimating unit 150 93 
estimates the waveform of the signal of the actual world 1 
by computing the equation supplied from the equation 
generating unit 15092. In other words, the actual world 

15 waveform estimating unit 15093 finds the approximation 
function as an approximation model, and supplies the 
approximation function to the image generating unit 15084 as 
the estimation result of the signal waveform of the actual 
world 1. Note that approximation function approximating the 

20 actual world 1 signals include functions with constant 
function values, regardless of argument value. 

The image generating unit 15084 recognizes the 
processing region in the input image from the processing 
region information supplied from the processing regions 

25 setting unit 15081. Also, the image generating unit 15084 
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generates a signal more closely approximating the signal of 
the actual world 1, from the approximation function supplied 
from the actual world waveform estimating unit 15093, that 
is to say, generates an image with no movement blurring, and 
5 replaces the image of the processing region of the input 

image with an output image wherein the image has no movement 
blurring, generates this as an output image, and supplies 
this to the image display unit 15085. 

Next, the processing of the signal processing device 4 

10 in Fig. 135 will be described with reference to the 
flowchart in Fig. 13 6. 

First, in step S15081, the signal processing device 4 
performs pre-processing, and the flow proceeds to step 
S15082 . That is to say, the signal processing device 4 

15 supplies one frame or one field, for example, of the input 

image, supplied from the sensor 2 (Fig. 1) as data 3, to the 
processing region setting unit 15081, continuity setting 
unit 15082, actual world estimating unit 15083, image 
generating unit 15084, and image display unit 15085. 

20 Further, the signal processing unit 4 causes the image 
display unit 15085 to display the input image. 

In step S15082, the user I/F 15086 determines whether 
or not there has been some sort of user input, by the user 
operating the user I/F 15086. In step S15082, in the event 

25 that determination is made that there is no user input, i.e., 
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in the event that the user has made no operations, the flow 
skips steps S15083 through S15085, and proceeds to step 
S15086 . 

On the other hand, in the event that determination is 
5 made in step S15082 that there has been user input, i.e., in 
the event that the user has viewed the input image displayed 
on the image display unit 15085 and operated the user I/F 
15086, thereby making user input indicating some sort of 
instruction or information, the flow proceeds to step S15083, 
10 where the user I/F 15086 determines whether or not the user 
input is ending instructions for instructing ending of the 
processing of the signal processing device 4. 

In the event that determination is made in step S15083 
that the user input is ending instructions, the signal 
15 processing device 4 ends processing. 

On the other hand, in the event that determination is 
made in step S15083 that the user input is not ending 
instructions, the flow proceeds to step S15084, where the 
user I/F 15086 determines whether or not the user input is 
2 0 constraint condition information. In the event that 

determination is made in step S15084 that the user input is 
not constraint condition information, the flow skips step 
S15085, and proceeds to step S15086. 

Also, in step S15084, in the event that the user input 
25 is determined to be constraint condition information, the 
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flow proceeds to step S15085, and the user I/F 15086 
supplies the constraint condition information to the 
equation generating unit 15092, and the flow proceeds to 
step S15086. 

5 In step S15086, the processing region setting unit 

15081 sets the processing region based on the input image, 
and supplies the processing region information identifying 
the processing region to the continuity setting unit 15082, 
model generating unit 15091 and equation generating unit 

10 15092 of actual world estimating unit 15083, and image 

generating unit 15084, and the flow proceeds to step S15087. 
Here, in the setting of processing region in step S15086 the 
user inputs the processing region instruction information by 
operating the user I/F 15086, and this can be performed 

15 based on the processing region instruction information, or 
it can be performed without processing region instruction 
information from the user. 

In step S15087, the continuity setting unit 15082 
recognizes the processing region in the input image, from 

20 the processing region information supplied from the 
processing region setting unit 15081. Further, the 
continuity setting unit 15082 sets the continuity of the 
actual world 1 signals lost from one portion of the 
continuity in the image data at that processing region, and 

25 supplies continuity information representing that continuity 
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to the model generating unit 15091 of the actual world 
estimating unit 15083 and the flow proceeds to step S15088. 

Here the continuity setting unit 15082 sets the 
movement amount v, expressing that the movement is in the 
5 horizontal direction at a constant speed of v pixels per 

shutter time (exposure time) , as the continuity information, 
and supplies this to the model generating unit 15091 of the 
actual world estimating unit 15083 . The continuity setting 
unit 15 0 82 sets the movement amount showing only the 

10 movement size as continuity information, under the premise 
that the object is moving in a horizontal direction, but a 
movement vector showing the size and direction of the 
movement of the object can be set as continuity information 
as well. The setting of continuity in step S15087 can be 

15 arranged by the user inputting the continuity instruction 
information by operating the user I/F 15086, and can be 
performed based on this continuity instruction information, 
or can be performed with no continuity instruction 
information from the user. 

20 The actual world estimating unit 15083 performs actual 

world estimating processing in step S15008. In other words, 
with the actual world estimating unit 15083, the model 
generating unit 15091 generates an equation (model equation) 
as the model (relation model) wherein the relation of the 

2 5 pixel values of each of the pixels of the input image which 
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has movement blurring and the signal of the actual world 1 
is modeled, based on the movement amount v supplied from the 
continuity setting unit 15082 in step S15087 and the 
processing region information supplied from the processing 
5 region setting unit 15081 in step S15086, and the generated 
model equation is supplied to the equation generating unit 
15092 . 

The equation generating unit 15092 generates a 
constraint condition expression based on the constraint 

10 condition information supplied from the user I/F 15086 in 
step S15085, and sets the weighting for adjusting the 
balance with the model equation in the generated constraint 
condition expression. Also, the equation generating unit 
15092 generates an equation for finding the approximation 

15 function as the model (approximation model) wherein the 

signal of the actual world 1 is modeled, from the weighted 
constraint condition expression and the model equation 
supplied from the model generating unit 15091 to the 
equation generating unit 15092, and substitutes the pixel 

2 0 values of each of the pixels of the input image into this 
equation, and supplies this to the actual world waveform 
estimating unit 15093. In the event that the user has not 
specified (set) constraint condition information, that is to 
say, in the event that NO is determined to be the case in 

25 step S15082 or S15084, the equation generating unit 15092 
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performs predetermined processing such as a method for 
solving a simultaneous equation formed of a model equation 
and an equation hypothesizing that the edge portions of the 
processing region is "flat", as described in the embodiment 
5 in Fig. 113, for example. 

Now, the weighting can be set for the constraint 
condition expression only, for the model equation only, or 
for both the constraint condition expression and the model 
equation. 

10 The actual world waveform estimating unit 15093 

estimates the waveform of the actual world 1 signal, that is 
to say, finds the approximation function as a model wherein 
the actual world 1 signal is modeled, by computing the 
equation supplied from the equation generating unit 15092, 

15 and the approximation function is supplied to the image 

generating unit 15084 as the estimation result of the signal 
waveform of the actual world 1. 

The details of the actual world estimating process of 
step S15088 will be described later with reference to Fig. 

20 138. 

After the processing in step S15088, the flow proceeds 
to step S15089, and the image generating unit 15084 
generates a signal more closely approximating the. signal of 
the actual world 1, from the approximation function supplied 
25 from the actual world waveform estimating unit 15093, that 
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is to say, generates an image with no movement blurring, and 
based on the processing region information supplied from the 
processing region setting unit 15081, converts the portion 
of the processing region of the input image into an image 
5 which has no movement blurring generated, generates this as 
an output image, and supplies this to the image display unit 
15085, and the flow proceeds to step S15090. 

In step S15090, the image display unit 15085 displays 
the output image supplied from the image generating unit 
10 15084 instead of the input image displayed in step S15081, 

or along with the input image, and the flow proceeds to step 
S15091 . 

In step S15091, as with the case of step S15082, the 
user I/F 15086 determines whether or not there has been some 

15 sort of user input by the user operating the user I/F 15086, 
and in the event that determination is made that there has 
been no user input, i.e., in the event that the user has 
made no operations, the flow returns to step S15091, and 
awaits some sort of user input . 

20 Also, in the event that determination is made in step 

S15091 that there has been user input, i.e., in the event 
that the user has viewed the input image or output image 
displayed on the image display unit 15085 and operated the 
user I/F 15086, thereby making user input representing some 

25 sort of instructions or information, the flow proceeds to 
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step S15092, where the user I/F 15086 determines whether or 
not the user input is ending instructions instructing ending 
of the processing of the signal processing device 4. 

In the event that determination is made in step S15092 
5 that the user input is ending instructions, the signal 
processing device 4 ends processing. 

On the other hand, in the event that determination is 
made in step S15092 that the user input is not ending 
instructions, the flow proceeds to step S15 0 93, where the 

10 user I/F 15086 determines whether or not the user input is 
constraint condition information. In the event that 
determination is made in step S15093 that the user input is 
not constraint condition information, the flow returns to 
step S15091, and the same processing is repeated thereafter. 

15 Also, in step S15093, in the event that determination 

is made that the user input is constraint condition 
information, the flow returns to step S15085, and as 
described above, the user I/F 15086 supplies the constraint 
condition information to equation generating unit 15092 . 

20 The flow then proceeds from step S15085 to step S15086, and 
hereafter the same processing is repeated. Now, in the 
event that the signal processing device 4 repeats the 
processing of step S15085 through step S15093, the 
processing of the above-described step S15086 and S15087 are 

25 the same as that set by the first processing S15086 and 
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S15087, and the same process as the first process may be 
repeated, or may be skipped. 

Thus, with the signal processing device 4 in Fig. 135, 
by finding an approximation function of a model wherein the 
5 signal of the actual world 1 is modeled, an output image of 
high image quality with no movement blurring can be 
generated. 

Also, by operating the user I/F 15086, the user can set 
constraint conditions which constrain the relation between 

10 each of the pixels in the original image having no movement 
blurring, and in the event that a new constraint condition 
is input, an image without movement blurring can be obtained 
again, and therefore an output image of high image quality 
according to the preference of the user can be easily 

15 acquired . 

In the signal processing device 4 in Fig. 135, an 
output image without movement blurring is generated by 
finding an approximation function which approximates the 
light signal of the actual world 1, but this output image 

2 0 can be regarded as having the movement blurring removed from 
the input image. Accordingly, it can be said that 
processing is performed for movement blurring removal with 
the signal processing device 4 in Fig. 135. 

Next, the internal configuration of the actual world 

25 estimating unit 15083 in Fig. 135 will be described with 
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reference to Fig. 137. With the embodiment in Fig. 135 also, 
the mechanism for generating movement blurring of objects 
reflecting in the input image is modeled as a relation of 
the above-described Expression (141) , which is an example 
5 shown in Figs. 125 and 126. 

In other words, the pixel values of each of the pixels 
of the movement blurring image as the input image are values 
wherein the pixel values (or the electric charge 
corresponding thereto) of each of the pixels of an image 

10 with no movement blurring which accumulate (integrate) while 
moving at the movement amount v, and models the mechanism 
for movement blurring generating. 

The actual world estimating unit 15083 comprises a 
model generating unit 15091, an equation generating unit 

15 15092, and an actual world waveform estimating unit 15093. 
Further, the equation generating unit 15092 comprises a 
constraint condition setting unit 15101, a weighting 
changing unit 15012, and a normal equation generating unit 
15103 . 

2 0 The movement amount v as the continuity information is 

supplied to the model generating unit 15091 by the 
continuity setting unit 15082 . The movement amount v in 
this example expresses the continuity of the object moving 
in the horizontal direction at a constant speed of v pixels 

25 per shutter time, within the processing region of the input 
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image which is set in the processing region setting unit 
15081, and for example, v = 4, as with the above -described 
example. Also, the movement amount v may be a value which 
differs for each line, and in the event that the movement 
5 amount v differs for each line, k number, equal to the 
number of lines, of the movement amount v is supplied as 
continuity information from the continuity setting unit 
15082 to the model generating unit 15091. 

Also, processing region information which specifies the 

10 processing region within the input image is also supplied to 
the model generating unit 15091 from the processing region 
setting unit 15081. In the case of using a rectangular- 
shaped region as the processing region, the processing 
region information can be expressed, for example, by 

15 coordinate values at opposite angles of a rectangular- shaped 
processing region as to an absolute coordinates system 
wherein a predetermined position (pixel) of the input image 
is the origin, or the center coordinate of the processing 
region and the width (horizontal direction) and the height 

20 (vertical direction) of the processing region. 

The model generating unit 15091 acquires, for example, 
a processing region width of 1, and k line numbers, as 
illustrated in Fig. 119, from the processing region 
information supplied from the processing region setting unit 

25 15081. In this example, let us say that the processing 
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region 15042 is rectangular-shaped, and for example, has 1 = 
10 and k - 4. In this case, the processing region width of 
each line (the number of pixels) are the same, but with the 
processing region 15042, each line may have a different 
5 processing region width. In the case that the processing 
region width 1 differs for each line, the model generating 
unit 15091 acquires a processing region width 1 of the same 
number (k) as the number of lines. 

Also, the model generating unit 15091 generates an 

10 equation (model equation) of a mixed pixel value Pi which is 
shown in the above -described Expression (141) , based on the 
movement amount v supplied from the continuity setting unit 
15012 and the processing region width 1 and k number of 
lines acquired from the processing region information, and 

15 supplies this to the normal equation generating unit 15103. 

The constraint condition information specified by the 
user operating the user I/F 15086 is supplied to the 
constraint condition setting unit 15101 from the user I/F 
15086. For example, the user performs an operation with the 

20 user I/F 15086 to select (specify) the desired constraint 

condition from the constraint conditions which constrain the 
relation of the pixel values Q h of each of the pixels of an 
image with no movement blurring, such as the above-described 
Expression (143) or Expression (144) , which have been set in 

25 advance inside the constraint condition setting unit 15101, 
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and the user I/F 15086 supplies the information describing 
the selected constraint conditions as the constraint 
condition information to the constraint condition setting 
unit 15101. 

5 The constraint condition setting unit 15101 selects a 

constraint condition expression corresponding to the 
constraint condition information supplied from the user I/F 
15086 from the constraint condition expressions within the 
constraint condition setting unit 15101 itself. Further, 

10 the constraint condition setting unit 15101 acquires the 
processing region width 1 based on the processing region 
information of the input image supplied from the equation 
generating unit 15092 from the processing region setting 
unit 15081, and generates only the number which corresponds 

15 to the processing region width 1, for the constraint 

condition expressions. For example, in the event that 
Expression (143) is used which has the condition "adjacent 
pixel difference = 0 " as the constraint condition 
information, the constraint condition setting unit 15101 

2 0 generates the constraint condition expression expressed by 
the above-described Expression (145) , by acquiring the 
processing region width 1 based on the processing region 
information supplied from the processing region setting unit 
15081, and supplies this to the normal equation generating 

25 unit 15103. In the example of the processing region 15042 
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illustrated in Fig. 119, the processing region 15042 is 
rectangular-shaped, and the processing region width 1 is the 
same of each line, but in the case that the processing 
region width 1 differs for each line, the number of 
constraint condition expressions also differ for each line. 

A weighting W m as to the model equation and a weighting 
W b as to the constraint condition expression are set in 
advance in the weighting changing unit 15102. With this 
embodiment, the weighting coefficient W relating to the 
constraint condition expression is set by Expression (153), 
and using only the weighting relating to the constraint 
condition expression, the balance of the model equation and 
the constraint condition expression is adjusted. 

w = w b /w m 

. . . (153) 

The weighting changing unit 15102 changes the weighting 
of the model equation and the constraint condition 
expression in the normal equation generated by the normal 
equation generating unit 15103, by supplying the weighting 
coefficient W to the normal equation generating unit 15103. 

The mixed pixel value Pi equation with the same number 
of k as the line numbers from the model generating unit 
15091 (Expression (141)) and the constraint condition 
Expression (Expression (145) ) corresponding to the 
constraint conditions selected from the constraint condition 
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setting unit 15101 by the user operating the user I/F 15086 
are supplied to the normal equation generating unit 15103 . 

With the normal equation generating unit 15103, the 
weighting coefficient W is set by the weighting changing 
5 unit 15102, regarding Expression (147) wherein the error e b j 
is considered in the constraint condition Expression (145) 
which is supplied from the constraint condition setting unit 
15101 (the weighting W as to the model equation and 
constraint condition [equation?] which was 1:1 is changed). 

10 In other words, Expression (14 7) is changed into the 

following Expression (154) of which both sides are 
multiplied by the weighting W. Here, the weighting 
coefficient W is for example a predetermined value such as 
0.1. In this case, a pixel value Q h , which makes the error 

15 e m i of the model equation smaller than the error e bj of the 
constraint condition expression, is found. 
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... (154) 

The normal equation generating unit 15103 selects one 
predetermined line as the line of interest from the 
processing region 15042 as shown in Fig. 128. Then, the 
5 normal equation generating unit 15103 generates the normal 
equation expressed by the following Expression (155) , from 
the mixed pixel value Pi equation (model equation) expressed 
by Expression (141) which is supplied from the model 
generating unit 15091 corresponding to the selected line of 
10 interest, and the constraint condition expression expressed 
by Expression (154) wherein the weighting coefficient W is 
added to Expression (14 5) which is supplied from the 
constraint condition setting unit 15101. 
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. . . (155) 

e m i is the error occurring in each expression in 
Expression (141) , and e b j is the error occurring in each 
expression in Expression (154) . 

Here, Expression (155) will be replaced by Expression 
(156) , using the matrix A 1 and the column vectors x, y, and 
e 1 . 

A'x = y+e' 

. . . (156) 

In this case, the total sum E 1 of the squared error of 
the error e m i of Expression (146) and the error e b j of 
Expression (154) can be expressed with the following 
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expression. 

E' =Ie rai 2 +W 2 Ze bj 2 

. . . (157) 

In order to find the Q h for which the total sum E of the 
squared error is the smallest, the column vector x (the 
column vector x with the pixel value Qh as a component) can 
be expressed by Expression (158) , from the equation similar 
to the above-described equations (150) and (151) . 

x= (A'VrVy 

. . . (158) 

Now, with Expression (158) , the superscript T means 
transposing, and the superscript -1 means an inverse matrix. 

With Expression (157) , when the weighting coefficient W 
is small, a sharp image can be found which can better 
satisfy Expression (141) even with some noise allowed, and 
on the other hand, when the weighting coefficient W is large, 
an image which is flat with little noise that can satisfy 
Expression (145) , rather than Expression (141) , can be found. 
That is to say, by adjusting the value of the weighting 
coefficient W, the balance of "sharpness" and "noise" can be 
adjusted . 

The normal equation generating unit 15103 computes the 
matrix (A ' T A ' ) _1 A ' T , which is a coefficient of the column 
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vector y of the right side of Expression (158) , and 
substitutes the mixed pixel value P± which is acquired based 
on the input image supplied from the sensor 2 (Fig. 1) into 
the column vector y. The matrix (A 1 T A ' ) _1 A 1 T is computed, 
5 and Expression (158) into which the mixed pixel value Pi is 
substituted is further supplied from the normal equation 
generating unit 15103 to the actual world waveform 
estimating unit 15093. 

When the normal equation generating unit 15103 finishes 

10 the above -described generation of the normal equation, the 
computation of the matrix (A 1 T A 1 ) X A 1 T , and the processing 
such as the substitution of the mixed pixel value Pi 
regarding the selected line of interest, a line not yet 
selected is selected as the next line of interest, and 

15 similar processing is performed as to the selected next line 
of interest. 

The actual world waveform estimating unit 15093, by 
computing Expression (158) which is supplied from the normal 
equation generating unit 15103, finds the column vector x, 
2 0 that is to say, the pixel value Q h with no movement blurring, 
and supplies this to the image generating unit 15084 (Fig. 
135) . 

The image generating unit 15084 replaces a portion of 
the processing region 15042 of the input image with the 
2 5 image of the pixel value Qh with no movement blurring, which 
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is supplied from the actual world waveform estimating unit 
15093, based on the processing region information supplied 
from the processing region setting unit 15081, and generates 
this as an output image, and supplies this to the image 
5 display unit 15085. 

Next, the actual world estimating processing of the 
actual world estimating unit 15083 in step S15088 in Fig. 
136 will be described with reference to the flowchart in Fig. 
138 . 

10 First, in step S15101, the model generating unit 15091 

acquires the processing region width 1 and the number of 
lines k, based on the processing region information supplied 
from the processing region setting unit 15081, and the flow 
proceeds to step S15102. 

15 In step S15102, the model generating unit 15091 

acquires the movement amount v, which is equivalent to the 
number of lines k in the processing region 15042, from the 
continuity setting unit 15082, and the flow proceeds to step 
S15103 . 

20 In step S15103, the model generating unit 15091 

generates an equation for the mixed pixel value Pi for each 
line, and the flow proceeds to step S15104 . In other words, 
the model generating unit 15091 generates k number of model 
equations (expressions (141) ) . 

25 In step S15104, the constraint condition setting unit 



- 419 - 

S04P0850 

15101 selects the constraint condition expression 
corresponding to the constraint condition information 
supplied from the user I/F 15086, from the constraint 
condition expressions existing within the constraint 

5 condition setting unit itself. Also, the constraint 

condition setting unit 15101 acquires the processing region 

width 1 from the processing region information of the input 

image which is supplied from the equation generating unit 

15092 from the processing region setting unit 15081, and 

10 generates an constraint condition Expression (Expression 

(145) ) corresponding to the constraint condition information 

from the user I/F 15086, which comprises a number of 

equations corresponding to the processing region width 1, 

and the flow proceeds to step S15105. 

15 In step S15105, the weighting changing unit 15102 

« 

supplies the weighting coefficient W expressed by Expression 
(153) , which is set in advance internally within the 
weighting changing unit 15102 itself, to the normal equation 
generating unit 15103, and changes the weighting as to the 
20 constraint condition expression, and the flow proceeds to 
step S15106. That is to say, the weighting changing unit 

15102 generates the above-described Expression (154) at the 
normal equation generating unit 15103. 

In step S15106, the normal equation generating unit 
25 15103 selects one predetermined line as the line of interest 
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from the processing region 15042, and the flow proceeds to 
step S15107. 

In step S15107, the normal equation generating unit 
15103 generates a normal equation expressed by Expression 
5 (155) (Expression (156) ) from Expression (146) , wherein the 
error e m i is considered of the various expressions of the 
equation of the mixed pixel value P± expressed by Expression 
(141) , which is supplied from the model generating unit 
15091, and the constraint condition Expression (154) 

10 obtained by changing the weighting of Expression (147) 

wherein the error e b j is considered regarding Expression 
(145) which is supplied from the constraint condition 
setting unit 15101, by the weighting changing unit 15102 in 
step S15105, and the flow proceeds to step S15108. 

15 In step S15108, the normal equation generating unit 

15103 computes the matrix (A ■ T A 1 ) _1 A ' T , which is a 
coefficient of the column vector y of the right side of 
Expression (158), and the flow proceeds to step S15109. 
In step S15109, the normal equation generating unit 

2 0 15103 substitutes the mixed pixel value Pi which is acquired 
based on the input image supplied from the sensor 2 (Fig. 1) 
into the column vector y, and the flow proceeds to step 
S15110. 

In step S15110, the actual world waveform estimating 
25 unit 15093 computes a normal equation. That is to say, the 
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actual world waveform estimating unit 15093 finds a pixel 
value Q h with no movement blurring by computing Expression 
(158) which is supplied from the normal equation generating 
unit 15103, and the flow proceeds to step S15111. 
5 In step S15111, determination is made as to whether or 

not processing is performed to find a pixel value Q h with no 
movement blurring as to all lines of the processing region 
15042. In the event that it is determined that processing 
is not performed to find a pixel value Q h with no movement 

10 blurring as to all lines of the processing region 15042, the 
process is returned to step S15106, and the processing in 
steps S15106 through S15111 is repeated. That is to say, a 
line not yet selected of the processing region 15042 is 
selected as the next line of interest, and processing is 

15 performed to find a pixel value Q h with no movement blurring. 
On the other hand, in the event that it is determined 
that processing is performed to find a pixel value Q h with 
no movement blurring as to all lines of the processing 
region 15042, the actual world estimating processing ends 

2 0 and returns . 

Thus, by generating a normal equation (Expression 
(155)) as to one line which is the line of interest of the 
processing region 15042, and repeating the processing to 
find a pixel value Q h with no movement blurring k number of 

25 times, which is the number of lines in the processing region 
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15042, the pixel value Q h with no movement blurring can be 
found across the entire region of the processing region 
15042 . 

In the event that the movement amount v and the 
5 processing region width 1 are each the same for each line of 
the processing region 15042, the matrix A in Expression 
(155) is the same for every line. Accordingly, the matrix 
(A 1 T A 1 ) _1 A 1 T in Expression (158) also is the same for every 
line, and the processing to compute the matrix (A |T A')~ 1 A' T 
10 for each line can be omitted. 

Next, the actual world estimating processing of the 
actual world estimating unit 15083 in step S15088 in Fig. 
13 6 in the event that the movement amount v and the 
processing region width 1 are each the same for each line of 
15 the processing region 15042 will be described with reference 
to the flowchart of Fig. 139. However, the portions in Fig. 
139 which are similar to the flowchart in Fig. 138 will omit 
the description thereof as appropriate. 

In Fig. 139, the processing in step S15138 which 
20 corresponds to step S15106 in Fig. 138 is positioned not 

between step S15135 and S15136, but between steps S15137 and 
S15139. Further, in step S15141, if it is determined that 
processing is not performed to find a pixel value Q h with no 
movement blurring as to all lines of the processing region 
25 15042, the process is returned to step S15138. 
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In other words, in the event that the movement amount v 
and the processing region width 1 of each line of the 
processing region 15042 are each the same, the normal 
equation generated in step S15136 is the same for each line, 
5 and therefore, after computing the matrix (A ,T A , )~ 1 A' T as to 
the first line of interest in step S15137, computing the 
matrix (A ,T A')" 1 A ,T as to the remaining lines of interest 
(the processing in step S15137) is not necessary. 
Accordingly, the selection of the line of interest (step 

10 S15138) , the substitution of the mixed pixel value P ± (the 
processing in step S1513 9) , and the computation of the 
normal equation (the processing in step S15140) only need be 
executed in order, as to the lines not selected in the 
processing region 15042. 

15 Thus, with the embodiment illustrated in Fig. 135, a 

normal equation is generated from an equation of the mixed 
pixel value P± wherein a pixel value generating movement 
blurring is modeled with a pixel value with no movement 
blurring (model equation) , and a constraint condition 

20 expression using the relation between adjacent pixels of a 

pixel value with no movement blurring, and by computing this 
normal equation, a pixel value with no movement blurring 
(approximation function) is found, and therefore an image 
with good image quality can be generated. 

25 Also, because the balance (weighting) of the mixed 
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pixel value Pi equation and the constraint condition 
expression can be adjusted, an image can be generated with 
no movement blurring, which considers "sharpness" and 
"noise" and has the image quality according to the 
5 preference of the user. 

With the above-described example, the constraint 
condition information supplied from the user I/F 15086 to 
the constraint condition setting unit 15101 is said to be 
information showing which constraint condition the user has 

10 selected (instructed) , out of the multiple constraint 
conditions previously set internally in the constraint 
condition setting unit 15101, but it may be set up such that 
that user can directly input the constraint condition 
expression by operating the user I/F 15086. 

15 With the embodiment of the signal processing device 4 

in Fig. 135, the weighting coefficient W relating to the 
constraint condition expression, which adjusts the weighting 
(balance) of the mixed pixel value Pi equation and the 
constraint condition expression, has been a predetermined 

20 value which is set in advance (0.1 in the above case) . 

However, setting the weighting coefficient W relating 
to the constraint condition expression as a predetermined 
value (fixed value) is to apply the constraint conditions as 
to the places (relation between pixels) wherein the 

25 constraint condition is not established, similar to the 
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places wherein the constraint condition is established, 
which causes deterioration of the processing image in some 
cases . 

Therefore, next, an embodiment will be described 
5 wherein the signal processing device 4 can determine the 
weighting relating to the constraint condition expression 
and so forth corresponding to the activity (features) of the 
input image . 

Fig. 14 0 is a configuration example of another 

10 embodiment with an application example of the signal 
processing device 4 illustrated in Fig. 111. 

An image of, for example, one frame or one field as the 
data 3 from the sensor 2 is input into the signal processing 
device 4 in Fig. 14 0. Here, an object having a 

15 predetermined shape is displayed in the input image, such 
object moving in the horizontal direction (crosswise 
direction) of the image at a constant speed of v pixels per 
shutter time (exposure time) . In other words, regarding the 
input image, the object is moving in a horizontal direction 

20 at the movement amount of v pixels, and therefore because of 
the time integration effect of the sensor 2, the light 
signal of the object and the light signal of the portion 
other than the object are mixed (time mixture) , and thus, 
the image is blurred at portions such as the borders of the 

25 object. With the signal processing device 4 in Fig. 140, a 
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high-quality output image is generated which has removed 
movement blurring from the input image what had occurred due 
to such time mixture. 

In Fig. 140, the processing region setting unit 15111, 
5 the continuity setting unit 15112, the actual world 

estimating unit 15113, image generating unit 15114, image 
display unit 15115, and user I/F 15116, each correspond to 
the processing region setting unit 10001, the continuity 
setting unit 10002, the actual world estimating unit 10003, 

10 image generating unit 10004, image display unit 10005, and 
user I/F 10006 shown in Fig. Ill, and basically perform the 
same processing as the processing region setting unit 10001, 
the continuity setting unit 10002, the actual world 
estimating unit 10003, image generating unit 10004, image 

15 display unit 10005, and user I/F 10006. Further, in Fig. 
140, the actual world estimating unit 15113 comprises a 
model generating unit 15121, an equation generating unit 
15122, and actual world waveform estimating unit 15123. The 
model generating unit 15121, equation generating unit 15122, 

20 and actual world waveform estimating unit 15123 correspond 
to each of the model generating unit 10011, equation 
generating unit 10012, and actual world waveform estimating 
unit 10013 in Fig. Ill, and basically perform the same 
functions as each of the model generating unit 10011, 

25 equation generating unit 10012, and actual world waveform 
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estimating unit 10013. 

Note however, that in Fig. 140, the assisting 
information which the user I/F 15116 outputs by the user 
operating the user I/F 15116, is supplied to the equation 
5 generating unit 15122. 

In other words, in Fig. 140, by operating the user I/F 
15116, with the equation generating unit 15122, the user can 
set the constraint conditions which constrain the relation 
between each of the pixels of an image with no movement 

10 blurring, which is an image of the actual world 1 

corresponding to the input image, and when the user performs 
an operation to set the constraint conditions, the user I/F 
15116 supplies the constraint condition information 
expressing the constraint conditions set by the operation to 

15 the equation generating unit 15122. 

Specifically, for example, a user who views an image 
displayed on the image display unit 15115 reflecting a 
blurring image guesses the original image, that is to say, 
an image without movement blurring which is an image of the 

20 actual world 1 corresponding to the input image. Also, for 
example, the user guesses predetermined regions where 
blurring is occurring as the edge portions, and guesses the 
relation of the pixel values between each of the pixels (the 
constraint conditions) , such as the difference being large 

2 5 between the pixel values (level) between the adjacent pixels 
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at the edge portions. Further, the user can set the 
conditions suitable for the guessed relation of the pixels 
value between each of the pixels into the equation 
generating unit 15122, by operating the user I/F 15116. 
5 Also, in Fig. 140, the processing region information, 

which is supplied from the processing region setting unit 
15111 to the actual world estimating unit 15113, is supplied 
to the model generating unit 15121 and the equation 
generating unit 15122 of the actual world estimating unit 

10 15113. The model generating unit 15121 and the equation 

generating unit 15122 recognize the processing region in the 
input image from the processing region information supplied 
from the processing region setting unit 15111. 

The continuity setting unit 15112 sets a movement 

15 amount (movement vector) v in the input image as the 

continuity information that an object is moving in the 
horizontal direction of the image at the constant speed of v 
pixels per shutter time, and supplies this movement amount v 
as the continuity information to the model generating unit 

20 15121 of the actual world estimating unit 15113. 

The model generating unit 15121 generates an equation 
(hereafter referred to as model equation as appropriate) as 
a model (hereafter referred to as relation model as 
appropriate) , modeling the relation between the signal of 

25 the actual world 1 and the pixel values of each of the 
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pixels of the movement blurring image as the input image 
which considers the movement amount v as the continuity 
information supplied from the continuity setting unit 15112, 
regarding the processing region recognized by the processing 
5 region information which is supplied from the processing 

region setting unit 15111, and supplies the generated model 
equation to the equation generating unit 15122. 

The equation generating unit 15122 generates a 
constraint condition Expression (hereafter referred to as 

10 constraint condition expression as appropriate) , based on 

the constraint condition information supplied from the user 
I/F 15116. Also, the equation generating unit 15122 sets 
the weighting to be determined by the activity of the input 
image into the constraint condition expression and the model 

15 equation supplied from the model generating unit 15121, and 
substitutes the pixel values of each of the pixels of the 
input image comprising the processing region into the 
equation formed from the weighted constraint condition 
expression and the model equation, and thus, generates an 

2 0 equation to find the approximation function as a model 

(hereafter referred to as approximation model) wherein the 
signal of the actual world 1 is modeled, and supplies this 
to the actual world waveform estimating unit 15123 . 
The actual world waveform estimating unit 15123 

2 5 estimates the waveform of the actual world 1 signals, by 
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computing the equation supplied from the equation generating 
unit 15122. In other words, the actual world waveform 
estimating unit 15123 finds the approximation function as an 
approximation model, and supplies the approximation function 
5 to the image generating unit 15114 as the estimation result 
of the signal waveform of the actual world 1 . Note that 
approximation function approximating the actual world 1 
signals include functions with constant function values, 
regardless of argument value. 

10 The image generating unit 15114 recognizes the 

processing region in the input image from the processing 
region information supplied from the processing region 
setting unit 15111. Also, the image generating unit 15114 
generates a signal more closely approximating the signal of 

15 the actual world 1, from the approximation function supplied 
from the actual world waveform estimating unit 15123, that 
is to say, generates an image with no movement blurring, and 
replaces the image of the processing region of the input 
image with an image wherein the image has no movement 

20 blurring, generates this as an output image, and supplies 
this to the image display unit 15115. 

Next, the processing of the signal processing device 4 
in Fig. 140 will be described with reference to the 
flowchart in Fig. 141. 

25 First, in step S15161, the signal processing device 4 
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performs pre-processing, and the flow proceeds to step 
S15162. That is to say, the signal processing device 4 
supplies one frame or one field, for example, of the input 
image, supplied from the sensor 2 (Fig. 1) as data 3, to the 
5 processing region setting unit 15111, continuity setting 
unit 15112, actual world estimating unit 15113, image 
generating unit 15114, and image display unit 15115. 
Further, the signal processing unit 4 causes the image 
display unit 15115 to display the input image. 

10 In step S15162, the user I/F 15116 determines whether 

or not there has been some sort of user input, by the user 
operating the user I/F 15116. In step S15162, in the event 
that determination is made that there is no user input, i.e., 
in the event that the user has made no operations, the flow 

15 skips steps S15163 through S15165, and proceeds to step 
S15166 . 

On the other hand, in step S15162, in the event that 
determination is made that there has been user input, i.e., 
in the event that the user has viewed the input image 

2 0 displayed on the image display unit 15115 and operated the 
user I/F 15116, thereby making user input indicating some 
sort of instruction or information, the flow proceeds to 
step S15163, where the user I/F 15116 determines whether or 
not the user input is user instructions instructing ending 

25 of the processing of the signal processing device 4. 
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In the event that determination is made that in step 
S15163 that the user input is ending instructions, the 
signal processing device 4 ends the processing. 

On the other hand, in the event that determination is 
5 made in step S15163 that the user input is not ending 

instructions, the flow proceeds to step S15164, where the 
user I/F 15166 determines whether or not the user input is 
constraint condition instruction information. In the event 
that determination is made in step S15164 that the user 
10 input is not constraint condition instruction information, 

step S15165 is skipped, and the flow proceeds to step S15166. 

Also, in step S15164, in the event that the user input 
is determined to be constraint condition information, the 
flow proceeds to step S15165, and the user I/F 15116 
15 supplies the constraint condition information to the 

equation generating unit 15122, and the flow proceeds to 
step S15166. 

In step S15166, the processing region setting unit 
15111 sets the processing region based on the input image, 

20 and supplies the processing region information identifying 
the processing region to the continuity setting unit 15112, 
the model generating unit 15121 and equation generating unit 
15122 of the actual world estimating unit 15113, and image 
generating unit 15114, and the flow proceeds to step S15167. 

2 5 Now, an arrangement maybe made wherein the setting of the 
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processing region in step S15166 is performed by the user 
inputting processing region instruction information by- 
operating the user I/F 15116, so as to be carried out based 
- on the processing region instruction information, or, this 
5 may be carried out without processing region instruction 
information from the user. 

In step S15167, the continuity setting unit 15112 
recognizes the processing region in the input image, from 
the processing region information supplied from the 

10 processing region setting unit 15111. Further, the 

continuity setting unit 15112 sets continuity of a part of 
the continuity of the actual world 1 signals that has been 
lost in the image data of the processing region, and 
supplies continuity information indicating the continuity 

15 thereof to the model generating unit 15121 of the actual 

world estimating unit 15113, and the flow proceeds to step 
S15168. Here, the continuity setting unit 15112 sets the 
movement amount v, expressing that the movement is in the 
horizontal direction at a constant speed of v pixels per 

2 0 shutter time (exposure time) , as the continuity information, 
and supplies this to the model generating unit 15121 of the 
actual world estimating unit 15113. Here, the continuity 
setting unit 15112 sets the movement amount showing only the 
movement size as continuity information, under the premise 

25 that the object is moving in a horizontal direction, but a 
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movement vector showing the size and direction of the 
movement of the object can be set as continuity information 
as well. Now, an arrangement may be made wherein the 
setting of the continuity in step S15167 is performed by the 
user inputting continuity instruction information by 
operating the user I/F 15116, so as to be carried out based 
on the continuity instruction information, or, this may be 
carried out without continuity instruction information 

The actual world estimating unit 15113 performs actual 
world estimating processing in step S15168. In other words, 
with the actual world estimating unit 15113, the model 
generating unit 15121 generates an equation (model equation) 
as the model (relation model) wherein the relation of the 
pixel values of each of the pixels of the input image which 
has movement blurring and the signal of the actual world 1 
is modeled, based on the movement amount v supplied from the 
continuity setting unit 15112 in step S15167 and the 
processing region information supplied from the processing 
region setting unit 15111 in step S15166, and the generated 
model equation is supplied to the equation generating unit 
15122. 

The equation generating unit 15122 generates a 
constraint condition expression based on the constraint 
condition information supplied from the user I/F 15116 in 
step S15165, and sets the weighting determined from the 
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activity of the input image into the generated constraint 
condition expression and the model equation supplied from 
the model generating unit 15121 to the equation generating 
unit 15122. Then, the equation generating unit 15122 
5 substitutes the pixel values of each of the pixels of the 
input image comprising the processing region into the 
equation formed from the weighted constraint condition 
expression and the model equation, and thus generates the 
equation for finding the approximation function as a model 

10 (approximate model) wherein the signal of the actual world 1 
is modeled, and supplies this to the actual world waveform 
estimating unit 15123. In the event that the user has not 
specified (set) constraint condition information, that is to 
say, in the event that NO is determined to be the case in 

15 step S15162 or S15164, the equation generating unit 15122 
performs predetermined processing such as a method for 
solving a simultaneous equation formed of a model equation 
and an equation hypothesizing that the edge portions of the 
processing region is "flat 11 , as described in the embodiment 

2 0 in Fig. 113, for example. 

The actual world waveform estimating unit 15123 
estimates the waveform of the actual world 1 signals, by 
computing the equation supplied from the equation generating 
unit 15122, that is to say, the actual world waveform 

25 estimating unit 15123 obtains an approximation function 
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modeling the actual world 1 signals, and supplies the 
approximation function to the image generating unit 15114, 
as estimation results of the waveform of the actual world 1 
signals . 

5 The details of the actual world estimating processing 

in step S15168 will be described later with reference to Fig. 
143 . 

After the processing in step S15168, the flow proceeds 
to step S15169, and the image generating unit 15114 

10 generates a signal more closely approximating the signal of 
the actual world 1, from the approximation function supplied 
from the actual world waveform estimating unit 15123, that 
is to say, generates an image with no movement blurring, and 
based on the processing region information supplied from the 

15 processing region setting unit 15111, converts the portion 
of the processing region of the input image into an image 
which has no movement blurring generated, generates this as 
an output image, and supplies this to the image display unit 
15115, and the flow proceeds to step S15170. 

20 In step S15170, the image display unit 15115 displays 

the output image supplied from the image generating unit 
15114 instead of the input image displayed in step S15161, 
or along with the input image, and the flow proceeds to step 
S15171 . 

25 In step S15171, as with the case of step S15162, the 
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user I/F 15116 determines whether or not there has been some 
sort of user input by the user operating the user I/F 15116, 
and in the event that determination is made that there has 
been no user input, i.e., in the event that the user has 
5 made no operations, the flow returns to step S15171, and 
awaits some sort of user input. 

Also, in the event that determination is made in step 
S15171 that there has been user input, i.e., in the event 
that the user has viewed the input image or output image 

10 displayed on the image display unit 15115 and operated the 
user I/F 15116, thereby making user input representing some 
sort of instructions or information, the flow proceeds to 
step S15172, where the user I/F 15116 determines whether or 
not the user input is ending instructions instructing ending 

15 of the processing of the signal processing device 4. 

In the event that determination is made in step S15172 
that the user input is ending instructions, the signal 
processing device 4 ends the processing. 

On the other hand, in the event that determination is 

20 made in step S15172 that the user input is not ending 

instructions, the flow proceeds to step S15173, where the 
user I/F 15116 determines whether or not the user input is 
constraint condition information. In the event that 
determination is made in step S15173 that the user input is 

25 not constraint condition information, the flow returns to 
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step S15171, and hereafter the same processing is repeated. 

Also, in step S15173, in the event that determination 
is made that the user input is constraint condition 
information, the flow returns to step S15165, and as 
5 described above, the user I/F 15116 supplies the constraint 
condition information to the equation generating unit 15122. 
The flow then proceeds from step S15165 to step S15166, and 
hereafter the same processing is repeated. In the event 
that the signal processing device 4 repeats the processing 
10 of step S15165 through step S15173, the processing of the 
above-described step S15166 and S15167 is the same as that 
set by the first processing S15166 and S15167, and the same 
process as the first process may be repeated, or may be 
skipped . 

15 Thus, with the signal processing device 4 in Fig. 140, 

by finding an approximation function as a model wherein the 
signal of the actual world 1 is modeled, an output image of 
high image quality with no movement blurring can be 
generated . 

20 Also, by operating the user I/F 15116, the user can set 

constraint conditions which constrain the relation between 
each of the pixels in the original image having no movement 
blurring, and in the event that a new constraint condition 
is input, an image without movement blurring can be obtained 

25 again, and therefore an output image of high image quality 
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according to the preference of the user can be easily 
acquired . 

With the signal processing device 4 in Fig. 140, an 
output image without movement blurring is generated by 
5 finding an approximation function which approximates the 
light signal of the actual world 1, but this output image 
can be regarded as having the movement blurring removed from 
the input image. Accordingly, it can be said that 
processing is performed for movement blurring removal with 

10 the signal processing device 4 in Fig. 140. 

Next, the configuration of the inside of the actual 
world estimating unit 15113 in Fig. 140 will be described 
with reference to Fig. 142. With the embodiment in Fig. 140 
also, the mechanism for movement blurring generation of the 

15 object projected on the input image is modeled as the 
relation of the above-described Expression (141) , 
illustrated in Fig. 125 and Fig. 126 which is one example 
thereof . 

In other words, let us say that the mechanism for 
2 0 moment blurring generation is modeled assuming that the 

pixel values of each of the pixels of the movement blurring 
image as the input image are values wherein the pixel values 
(or the electric charge corresponding thereto) of each of 
the pixels of an image with no movement blurring which are 
25 accumulated (integrated) while moving at the movement amount 
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v. 

The actual world estimating unit 15113 comprises a 
model generating unit 15121, an equation generating unit 
15122, and an actual world waveform estimating unit 15123. 
5 Further, the equation generating unit 15122 comprises a 
constraint condition setting unit 15131, an activity 
detecting unit 15132, a weighting changing unit 15133, and a 
normal equation generating unit 15134. 

The movement amount v as the continuity information is 

10 supplied to the model generating unit 15121 by the 

continuity setting unit 15112. The movement amount v in 
this example expresses the continuity of the object moving 
in the horizontal direction at a constant speed of v pixels 
per shutter time, within the processing region of the input 

15 image which is set in the processing region setting unit 

15111, and for example, let us say that v = 4, as with the 
above-described example. Also, the movement amount v may be 
a value which differs for each line, and in the event that 
the movement amount v differs for each line, k number, equal 

2 0 to the number of lines, of the movement amount v is supplied 
as continuity information from the continuity setting unit 
15112 to the model generating unit 15121. 

Also, processing region information which specifies the 
processing region within the input image is also supplied to 

25 the model generating unit 15121 from the processing region 
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setting unit 15111. In the case of employing a rectangular- 
shaped region as the processing region, the processing 
region information can be expressed, for example, by 
coordinate values at opposite angles of a rectangular- shaped 
5 processing region as to an absolute coordinate system 

wherein a predetermined position (pixel) of the input image 
is the origin, or the center coordinate of the processing 
region and the width (horizontal direction) and the height 
(vertical direction) of the processing region. 

10 The model generating unit 15121 acquires, for example, 

a processing region width of 1, and k line numbers, as 
illustrated in Fig. 119, from the processing region 
information supplied from the processing region setting unit 
15111. In this example, let us say that the processing 

15 region 15042 is rectangular- shaped, and for example, has 1 = 
10 and k = 4. In this case, the processing region width of 
each line (the number of pixels) are the same, but with the 
processing region 15042, each line can have a different 
processing region width. In the case that the processing 

20 region width 1 differs for each line, the model generating 
unit 15121 acquires a processing region width 1 of the same 
number (k) as the number of lines. 

Also, the model generating unit 15121 generates an 
equation (model equation) of a mixed pixel value P± (i = 3 

25 through 9) which is shown in the above -described Expression 
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(141) , based on the movement amount v supplied from the 
continuity setting unit 15112 and the processing region 
width 1 and k number of lines acquired from the processing 
region information, and supplies this to the normal equation 
5 generating unit 15134. 

The constraint condition information specified by the 
user operating the user I/F 15116 is supplied to the 
constraint condition setting unit 15131 from the user I/F 
15116. For example, the user performs an operation with the 

10 user I/F 15116 to select (specify) the desired constraint 

condition from the constraint conditions which constrain the 
relation of the pixel values Q h of each of the pixels of an 
image with no movement blurring, such as the above -described 
Expression (143) or Expression (144) , which have been set in 

15 advance inside the constraint condition setting unit 15131, 
and the user I/F 15116 supplies the information describing 
the selected constraint conditions as the constraint 
condition information to the constraint condition setting 
unit 15131. 

20 The constraint condition setting unit 15131 selects a 

constraint condition expression corresponding to the 
constraint condition information supplied from the user I/F 
15116 from the constraint condition expressions within the 
constraint condition setting unit 15131 itself. Further, 

25 the constraint condition setting unit 15131 acquires the 
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processing region width 1 based on the processing region 
information of the input image supplied to the equation 
generating unit 15122 from the processing region setting 
unit 15111, and generates only the number, which corresponds 
5 to the processing region width 1, of the constraint 

condition expressions. For example, in the event that 
Expression (143) is employed which has the condition 
"adjacent pixel difference = 0 11 as the constraint condition 
information, the constraint condition setting unit 15131 

10 generates the constraint condition expression expressed by 
the above-described Expression (145) , by acquiring the 
processing region width 1 based on the processing region 
information supplied from the processing region setting unit 
15111, and supplies this to the normal equation generating 

15 unit 15134. In the example of the processing region 15042 
illustrated in Fig. 119, the processing region 15042 is 
rectangular-shaped, and the processing region width 1 is the 
same of each line, but in the case that the processing 
region width 1 differs for each line, the number of 

20 constraint condition expressions also differ for each line. 

With the activity detecting unit 15132, the activity 
(features) of each of the pixels (pixel of interest) within 
the processing region of the input image, which is to become 
information for the purpose of the weighting changing unit 

25 15133 to determine the weighting W mi as to the equation of 
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each of the mixed pixels Pi (model equation) and the 
weighting W b j as to each of the constraint condition 
expressions, is detected. 

In other words, the activity detecting unit 15132 
5 detects the value of the adjacent pixel difference (Pj - 
Pj + i) between the value of the pixel of interest Pj and the 
pixel Pj+i adjacent to the right thereof, or the like, for 
example, with each of the pixels within the processing 
region of the input image as a pixel of interest Pj , for 
10 example, as activity of the pixel of interest Pj , and 
supplies this to the weighting changing unit 15133. 

The weighting changing unit 15133 has a weighting W mi 
(hereafter will be simply called weighting W m i) as to .the 
equation for each of the mixed pixel values Pi (the model 
15 equation) and a weighting W b j (hereafter will be simply 

called weighting W bj ) as to each of the constraint condition 
expressions . 

By supplying the weighting W mi and the weighting W bj to 
the normal equation generating unit 15134, the weighting 

20 changing unit 15133 changes (sets) the weighting of each of 
the equations of the mixed pixel value Pi equation supplied 
from the model generating unit 15121 (the model equation) , 
and the equation formed from the constraint condition 
expressions supplied from the constraint condition setting 

25 unit 15131, which are generated in the normal equation 
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generating unit 15134. 

The deterioration of the image comprised by the pixel 
value Q h acquired with the actual world estimating unit 
15113 can be conceived to occur often in positions where the 
5 constraint condition is inappropriate, and thus, adjusting 
the weighting W b j appropriately is more effective than the 
weighting W mi . Thus, with this embodiment, the weighting W mi 
is a predetermined value (fixed value) , and the weighting W bj 
is changed based on the activity supplied from the activity 

10 detecting unit 15132. Now, the weighting W mi may also be 
made to be changed by the activity found with similar 
methods to be described below, or by other methods. 

In the event that the "adjacent pixel difference =0" 
expressed by the above-described Expression (143) , for 

15 example, is employed as a constraint condition expression, 
the positions (pixels) wherein the constraint condition 
expression expressed by Expression (143) is not established 
are pixels corresponding to the edge portions of the object 
in the image with no movement blurring. Here, the 

2 0 constraint condition expression expressed by Expression 

(143) considers the difference between adjacent pixels in 
the direction of the spatial direction X, and therefore, the 
edge portion of the object wherein the constraint condition 
expression is not established is the edge in the direction 

25 that the object edge direction (border line) is not in the 
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horizontal direction (a parallel direction to the spatial 
direction X) . 

The edge portion of the image with no movement blurring 
occurring is also the edge in an image with movement 
5 blurring. In other words, a pixel which was an edge portion 
in an image with no movement blurring occurring still has 
the characteristics of the edge in the vicinity of this 
pixel even in an image with movement blurring occurring. 
Accordingly, the weighting W bj can be determined according to 

10 the value of the adjacent pixel difference (Pj - Pj+i) of the 
image (input image) wherein movement blurring is occurring. 

In other words, positions with a large value of the 
adjacent pixel difference (Pj - Pj +i ) of the input image has 
a greater probability of being an edge in an image with no 

15 movement blurring occurring. Thus, the weighting W bj of the 
constraint condition expression in the above-described 
Expression (143) wherein the adjacent pixel difference is 
small is made to be smaller (the constraint condition is 
relaxed) . There are two methods as will be described below, 

2 0 for example, as methods for causing the weighting W b j to be 
smaller corresponding to the adjacent pixel difference (Pj - 
Pj+i) which shows activity. 

The first method is a method employing the inverse 
ratio of the value of the adjacent pixel difference (Pj - 

25 Pj+i) of the input image as the weighting W b j . This can be 



- 447 - 



S04P0850 



expressed as the following Expression. 




j + 1 



j =0. • ■ -.8 



(159) 



The second method is a method setting the weighting W bj 
of the constraint condition as either 0 or 1 , according to 
the value of the adjacent pixel difference (Pj - Pj+i) of the 
input image, and equivalently determining whether or not to 
include the constraint condition expression. This can be 
expressed as the following Expression. 



Here, in Expression (160) , the threshold value Th' is a 
value which is the basis for determining whether or not to 
include the constraint condition expression, and can be set 
in advance in the weighting changing unit 15133. The 
constraint condition expression wherein the weighting W bj is 
set as 0 is essentially removed from the condition 
constraint expression . 

The equation of the mixed pixel value Pi having the same 
number k as the number of lines (Expression (141) ) is 
supplied to the normal equation generating unit 15134 from 
the model generating unit 15121. Also, the constraint 
condition Expression (Expression (145)) corresponding to the 




(Pj-P j+1 >Th) 
(Pj-P j+1 <Th) 



j = 0, • • 8 



(160) 
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constraint condition selected by the user operating the user 
I/F 15116 is supplied to the normal equation generating unit 
15134 from the constraint condition setting unit 15132. 

With the normal equation generating unit 15134, the 
weighting W bj is set for Expression (14 7) wherein the error 
e b j is considered to the constraint condition Expression 
(145) supplied from the constraint condition setting unit 
15132, by the weighting changing unit 15133 (the weighting 
W b j as to the model equation and the constraint condition 
which was has been 1:1 is changed) . 

In other words, Expression (14 7) is changed to the 
following Expression (161) which has multiplied both sides 
thereof by W b j . 

W b0 (Q 0 -Qi) =W b0 e b0 
«ti (Oi-Qz) =W b ie b i 
W b2 (Q2-Q 3 ) =W b2 e b2 

W b3 (Q 3 -Q 4 ) =W b3 e b3 
W b4 (Q 4 -Q 5 ) = W b4 e b4 

W b5 (Q5-Q6) =W b5 e b5 
W b6 (Q 6 -Q 7 ) =W b6 e b6 

W b7 (Q 7 -Q 8 ) =W b7 e b7 

w b8 (Q8-Q 9 ) =W b8 e b 8 

. . . (161) 

Also, with the normal equation generating unit 15134, 
the weighting W m i is set for Expression (146) wherein the 
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error e m i is considered to the mixed pixel value Pi equation 
(model equation) (141) supplied from the model generating 
unit 15121, by the weighting changing unit 15133. 

In other words, Expression (14 6) is changed to the 
following Expression (162) which has multiplied both sides 
thereof by the weighting W m i . 





x (Qo+Qi+Qg+Qg) /v 


= W m3 x 




W m4 


x (Q 1 +Q 2 +Q 3 4-Q 4 )/v 


= W m4 x 


(P 4 +e m4 


VU 


x (Q 2 +Q3+Q4+Q 5 )/v 


= W m5 x 


(P5 + 6m5 


w m6 


x (Q 3 +Q 4 +Q 5 +Q 6 )/v 


= W m6 x 




w m7 


x (Q 4 +Q 5 +Q 6 +Q 7 )/v 


= W m7 x 


(P 7 +e m7 




x (Q 5 +Q 6 +Q 7 +Q 8 )/v 


= W m8 x 


(Ps+e^ 


w m9 


x (Q 6 +Q 7 +Q 8 +Q 9 )/v 


= W m9 x 


(P9 + e m9 



. . . (162) 

As described above, the weighting W mi in Expression 
(162) is a fixed value, here. 

As illustrated in Fig. 128, the normal equation 
generating unit 15134 selects one predetermined line from 
the processing region 15042 as the line of interest. Then 
the normal equation generating unit 15134 generates a normal 
equation expressed with the following Expression (163) from 
Expression (161) and Expression (162) corresponding to the 
selected line of interest. 
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. . . (163) 

Here, Expression (163) is replaced with Expression 
(164), using the matrix A" and the column vectors x, y, e" . 

A"x = y+e" 

. . . (164) 

In this case, the total sum E of the squared errors of 
the error e b j of Expression (161) and the error e mi of 
Expression (162) can be expressed with the following 
expression . 



E" = I(W rai e mi ) 2 + I(W bj e bj ) 2 
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In order to find the Q h which causes the total sum E" of 
the squared error to be very small, the column vector x (the 
column vector x with the pixel value Q h as a component 
5 thereof) can be expressed with Expression (166) from the 

same expression as the above Expression (150) and Expression 
(151) . 

x= (A ,,T A")" 1 A" T y 

. . . (166) 

10 Now, with Expression (166) , the superscript T means 

transposing, and the superscript -1 means an inverse matrix. 

Accordingly, the normal equation generating unit 15134 
computes the matrix (A l,T A n ) ~ 1 A" T , which is a coefficient of 
the column vector y of the right side of Expression (166) , 

15 and substitutes the mixed pixel value Pi which is acquired 

based on the input image supplied from the sensor 2 (Fig. 1) 
into the column vector y. The matrix (A" T A n ) " 1 A" T is 
computed, and Expression (166) into which the mixed pixel 
value Pi is substituted is supplied from the normal equation 

20 generating unit 15134 to the actual world waveform 
estimating unit 15123. 

When the processing such as described above is finished 
wherein one predetermined line of the processing region 
15042 is selected as the line of interest, the normal 
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equation generating unit 15134 selects a line not yet 
selected as the next line of interest, and similar 
processing is performed as to the selected next line of 
interest . 

5 The actual world waveform estimating unit 15123, by 

computing Expression (166) which is supplied from the normal 
equation generating unit 15134, finds the column vector x, 
that is to say, the pixel value Q h with no movement blurring, 
and supplies this to the image generating unit 15114 (Fig. 
10 140) . 

The image generating unit 15114 replaces a portion of 
the processing region 15042 of the input image with the 
image of the pixel value Q h with no movement blurring, which 
is supplied from the actual world waveform estimating unit 

15 15123, based on the processing region information supplied 

from the processing region setting unit 15111, and generates 
this as an output image, and supplies this to the image 
display unit 15115. 

Next, the actual world estimating processing of the 

20 actual world estimating unit 15113 in step S15168 in Fig. 

141 will be described with reference to the flowchart in Fig. 
143 . 

First, in step S15191, the model generating unit 15121 
acquires the processing region width 1 and the number of 
25 lines k, based on the processing region information supplied 
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from the processing region setting unit 15111, and the flow 
proceeds to step S15192. 

In step S15192, the model generating unit 15121 
acquires a k number of movement amount v, which is 
5 equivalent to the number of lines in the processing region 
15042, from the continuity setting unit 15112, and the flow 
proceeds to step S15193. 

In step S15193, the model generating unit 15121 
generates an equation for the mixed pixel value Pi for each 

10 line, and the flow proceeds to step S15194. In other words, 
the model generating unit 15121 generates k number of model 
equations (expressions (141)). 

In step S15194, the constraint condition setting unit 
15131 selects the constraint condition expression 

15 corresponding to the constraint condition information 
supplied from the user I/F 15116, from the constraint 
condition expressions existing within the constraint 
condition setting unit itself. Also, the constraint 
condition setting unit 15131 acquires the processing region 

2 0 width 1 from the processing region information of the input 
image which is supplied from the equation generating unit 
15122 from the processing region setting unit 15111, and 
generates an constraint condition Expression (Expression 
(145) ) corresponding to the constraint condition information 

25 from the users interface 15116, which comprises a number of 
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equations corresponding to the processing region width 1, 
and the flow proceeds to step S15195. 

In step S15195, the activity detecting unit 15132 
detects the activity (features) of each of the pixels within 
5 the processing region of the input image which is 

information for the weighting changing unit 15133 to 
determine the weighting W b j , and supplies this to the 
weighting changing unit 15133, and the flow proceeds to step 
S15196 . 

10 In step S15196, the weighting changing unit 15133 

calculates the weighting W bj to be found with, for example, 
the above described first or second method, using the 
activity of each of the pixels within the processing region 
of the input image, which is supplied from the activity 

15 detecting unit 15132. Also, the weighting changing unit 

15133 sets the calculated weighting W bj and the weighting W m ± 
provided in advance as a fixed value as the weighting 
respectively for each of the equations of the mixed pixel 
value Pi equation of the normal equation generating unit 

2 0 15134 (model equation) and the expression for constraint 

conditions (constraint condition expression) , and the flow 
proceeds to step S15197. 

In step S15197, the normal equation generating unit 

15134 selects one predetermined line from the processing 

25 region 15042 as a line of interest, and the flow proceeds to 
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step S15198. 

In step S15198, the normal equation generating unit 
15134 generates a normal equation expressed by Expression 
(163) (Expression (164) ), from Expression (162) wherein the 
5 weighting W mi is changed by the weighting changing unit 15133 
in step S15196, as to Expression (146) which considers the 
error e m i of each of the expressions in the equation of the 
mixed pixel value Pi expressed with Expression (141) , which 
is supplied from the model generating unit 15121, and the 

10 constraint condition Expression (161) wherein the weighting - 
W b j is changed by the weighting changing unit 1513 3 in step 
S15196, as to Expression (147) which considers the error e b j 
of each of the expressions in the constraint condition 
Expression (145) , which is supplied from the constraint 

15 condition setting unit 15131, and the flow proceeds to step 
S15199 . 

In step S15199, the normal equation generating unit 
15134 computes the matrix (A ,|T A n ) " 1 A" T , which is a 
coefficient of the column vector y of the right side of 
20 Expression (166), and the flow proceeds to step S15200. 

In step S152 00, the normal equation generating unit 
15134 substitutes the mixed pixel value Pi which is acquired 
based on the input image supplied from the sensor 2 (Fig. 1) 
into the column vector y, and the flow proceeds to step 
25 S15201. 
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In step S15201, the actual world waveform estimating 
unit 15123 computes a normal equation. That is to say, the 
actual world waveform estimating unit 15123 finds a pixel 
value Q h with no movement blurring by computing Expression 
5 (166) which is supplied from the normal equation generating 
unit 15134, and the flow proceeds to step S15202 . 

In step S15202, determination is made as to whether or 
not processing is performed to find a pixel value Qh with no 
movement blurring as to all lines of the processing region 

10 15042. In the event that it is determined that processing 
is not performed to find a pixel value Q h with no movement 
blurring as to all lines of the processing region 15042, the 
process is returned to step S15197, and the processing in 
steps S15197 through S15202 is repeated. That is to say, a 

15 line not yet selected of the processing region 15042 is 
selected as the next line of interest, and processing is 
performed to find a pixel value Q h with no movement blurring. 

On the other hand, in the event that it is determined 
that processing is performed to find a pixel value Q h with 

20 no movement blurring as to all lines of the processing 

region 15042, the actual world estimating processing ends 
and returns . 

Thus, by generating a normal equation (Expression 
(163)) as to one line which is the line of interest of the 
25 processing region 15042, and repeating the processing to 
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find a pixel value Qh with no movement blurring k number of 
times, which is the number of lines in the processing region 
15042, the pixel value Q h with no movement blurring can be 
found across the entire region of the processing region 
5 15042. 

Thus, with the embodiment illustrated in Fig. 140, a 
normal equation is generated from an equation of the mixed 
pixel value P± wherein a pixel value generating movement 
blurring is modeled with a pixel value with no movement 

10 blurring, and a constraint condition expression using the 
relation between adjacent pixels of a pixel value with no 
movement blurring, and by computing this normal equation, a 
pixel value with no movement blurring is found, and 
therefore an image with good image quality can be generated. 

15 Also, the weighting of each of the constraint 

conditions expressions is changed based on the activity of 
the input image, and therefore an image with good image 
quality can be generated according to the features of the 
input image . 

20 Figs. 144 through 146 are diagrams describing the 

processing results of the signal processing device 4 in Fig. 
140 . 

Fig. 144 illustrates an input image to be input in the 
signal processing device 4 in Fig. 140. With the input 
25 image in Fig. 144, the characters "TAL" are moving in the 
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direction from left to right at the same speed, and movement 
blurring is occurring in the horizontal direction. 

Fig. 145 illustrates an output image wherein the input 
image in Fig. 144 has been processed by the signal 
5 processing device 4 in Fig. 135. That is to say, this is an 
output image as to the input shown in Fig. 144, wherein the 
weighting W b j regarding the constraint conditions is applied 
to all constraint condition expressions as the same weight. 
As is clear by comparing Fig. 144 and Fig. 145, in the image 

10 in Fig. 145, the movement blurring occurring in the input 
image in Fig. 144 is reduced, and the characters 11 TAL " are 
relatively clearly displayed. 

On the other hand, Fig. 146 illustrates the output 
image wherein the input image in Fig. 144 has been processed 

15 by the signal processing device 4 in Fig. 140 which is the 

above -described embodiment, using the above-described second 
method. That is, this is an output image as to the input 
shown in Fig. 144, wherein the weighting W b j regarding the 
constraint condition expressions is set to 0 or to 1 

20 depending on the activity. In the image in Fig. 146 also, 
the movement blurring occurring in the input image in Fig. 
144 is reduced, and the characters "TAL." are clearly 
displayed. 

Now, with the embodiment in Fig. 144 through Fig. 146, 
25 the processed image in Fig. 146 is an image wherein the edge 
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portion is clearer than the processed image in Fig. 145. 

Also, in the above-described example, the constraint 
condition information supplied from the user I/F 15116 to 
the constraint condition setting unit 15131 is said to be 
5 information showing which constraint condition the user has 
selected (specified) from the multiple constraint conditions 
set in advance inside the constraint condition setting unit 
15131, but the user may directly input the constraint 
condition expression by operating the user I/F 15116. 

10 Also, in the case that the above -described second 

method is used with the weighting changing unit 15133, the 
threshold value Th which is a value that is a basis for 
determining whether or not to input a constraint condition 
expression is said to be set in advance internally, but the 

15 user can also adjust the threshold value Th as appropriate 

by operating the user I/F 15116 while viewing the processing 
image displayed on the image display unit 15115. Fig. 147 
illustrates a configuration example of another embodiment of 
an application example of the signal processing device 4 in 

20 Fig. 111. 

An image of, for example, one frame or one field as the 
data 3 from the sensor 2 is input into the signal processing 
device 4 in Fig. 147. Here, let us say that an object 
having a predetermined shape is displayed in the input image, 
25 such object moving in the horizontal direction (crosswise 
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direction) of the image at a constant speed of v pixels per 
shutter time (exposure time) . In other words, regarding the 
input image, the object is moving in the horizontal 
direction at the movement amount of v pixels, and therefore 
5 because of the time integration effect of the sensor 2, the 
light signal of the object and the light signal of the 
portion other than the object are mixed (time mixture) , and 
thus, the image is blurred at portions such as the borders 
of the object. Thus, with the signal processing device 4 in 

10 Fig. 147, an output image of high image quality with no 
movement blurring can be generated by such time mixture. 

In Fig. 147, the processing region setting unit 15151, 
the continuity setting unit 15152, the actual world 
estimating unit 15153, image generating unit 15154, image 

15 display unit 15155, and user I/F 15156, each correspond to 
the processing region setting unit 10001, the continuity 
setting unit 10002, the actual world estimating unit 10003, 
image generating unit 10004, image display unit 10005, and 
user I/F 10006 shown in Fig. Ill, and basically perform the 

20 same processing as the processing region setting unit 10001, 
the continuity setting unit 10002, the actual world 
estimating unit 10003, image generating unit 10004, image 
display unit 10005, and user I/F 10006. Further, in Fig. 
147, the actual world estimating unit 15153 comprises a 

25 model generating unit 15161, an equation generating unit 
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15162, and actual world waveform estimating unit 15163. The 
model generating unit 15161, equation generating unit 15162, 
and actual world waveform estimating unit 15163 correspond 
to each of the model generating unit 10011, equation 
5 generating unit 10012, and actual world waveform estimating 
unit 10013 in Fig. Ill, and basically perform the same 
functions as each of the model generating unit 10011, 
equation generating unit 10012, and actual world waveform 
estimating unit 10013. 

10 Note however, that in Fig. 147, the assisting 

information which the user I/F 15156 outputs by the user 
operating the user I/F 15156, is supplied only to the 
equation generating unit 15162 . 

In other words, in Fig. 147, by operating the user I/F 

15 15116, with the equation generating unit 15162, the user can 
set the constraint conditions which constrain the relation 
between each of the pixels of an image with no movement 
blurring, which is an image of the actual world 1 
corresponding to the input image and the weighting as to the 

2 0 constraint condition thereof, and when the user performs an 
operation to set the constraint conditions or an operation 
to set the weighting, the user I/F 15156 supplies the 
constraint condition information expressing the constraint 
conditions set by the operation or the weighting as to the 

25 constraint condition to the equation generating unit 15162. 
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Specifically, for example, a user who views an image 
displayed on the image display unit 15155 reflecting a 
blurring image guesses the original image, that is to say, 
an image without movement blurring which is an image of the 
5 actual world 1 corresponding to the input image. Also, for 
example, the user guesses predetermined regions where 
blurring is occurring as the edge portions, and guesses the 
relation of the pixel values between each of the pixels (the 
constraint conditions) , such as the difference being large 

10 between the pixel values (level) between the adjacent pixels 
at the edge portions. Further, the user can set the 
conditions suitable for the guessed relation of the pixels 
value between each of the pixels into the equation 
generating unit 15162, by operating the user I/F 15156. 

15 Also, in Fig. 147, the processing region information, 

which is supplied from the processing region setting unit 
15151 to the actual world estimating unit 15153, is supplied 
to the model generating unit 15161 and the equation 
generating unit 15162 of the actual world estimating unit 

20 15153. The model generating unit 15161 and the equation 

generating unit 15162 recognize the processing region in the 
input image from the processing region information supplied 
from the processing region setting unit 15151. 

The continuity setting unit 15152 sets a movement 

2 5 amount (movement vector) v in the input image as the 
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continuity information that an object is moving in the 
horizontal direction of the image at the constant speed of v 
pixels per shutter time, and supplies this movement amount v 
as the continuity information to the model generating unit 
5 15161 of the actual world estimating unit 15153. 

The model generating unit 15161 generates an equation 
(hereafter referred to as model equation as appropriate) as 
a model (hereafter referred to as relation model as 
appropriate) , modeling the relation between the signal of 

10 the actual world 1 and the pixel values of each of the 

pixels of the movement blurring image as the input image 
which considers the movement amount v as the continuity 
information supplied from the continuity setting unit 15152, 
regarding the processing region recognized by the processing 

15 region information which is supplied from the processing 

region setting unit 15151, and supplies the generated model 
equation to the equation generating unit 15162. 

The equation generating unit 15162 generates a 
constraint condition Expression (hereafter referred to as 

20 constraint condition expression as appropriate) , based on 

the constraint condition information supplied from the user 
I/F 15156. Also, the equation generating unit 15162 sets 
the weighting supplied from the user I/F 15156 into the 
constraint condition expression and the model equation 

25 supplied to the equation generating unit 15162 by the model 
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generating unit 15161, and substitutes the pixel values of 
each of the pixels of the input image comprising the 
processing region into the equation formed from the weighted 
constraint condition expression and the model equation, and 
5 thus, generates an equation to find the approximation 

function as a model (hereafter referred to as approximation 
model) wherein the signal of the actual world 1 is modeled, 
and supplies this to the actual world waveform estimating 
unit 15163. 

10 The actual world waveform estimating unit 15163 

estimates the waveform of the actual world 1 signal, by 
computing the equation supplied from the equation generating 
unit 15162. In other words, the actual world waveform 
estimating unit 15163 finds the approximation function as an 

15 approximation model, and supplies the approximation function 
to the image generating unit 15154 as the estimation result 
of the signal waveform of the actual world 1 . Note that 
approximation function approximating the actual world 1 
signals include functions with constant function values, 

2 0 regardless of argument value. 

The image generating unit 15154 recognizes the 
processing region in the input image, based on the 
processing region information supplied from the processing 
region setting unit 15151. Also, the image generating unit 

25 15154 generates a signal more closely approximating the 
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signal of the actual world 1, from the approximation 
function supplied from the actual world waveform estimating 
unit 15163, that is to say, generates an image with no 
movement blurring, and replaces the image of the processing 
5 region of the input image with an output image wherein the 
image has no movement blurring, generates this as an output 
image, and supplies this to the image display unit 15155. 

Next, the processing of the signal processing device 4 
in Fig. 147 will be described with reference to the 

10 flowchart in Fig. 148. 

First, in step S15211, the signal processing device 4 
performs pre-processing, and the flow proceeds to step 
S15212. That is to say, the signal processing device 4 
supplies one frame or one field, for example, of the input 

15 image, supplied from the sensor 2 (Fig. 1) as data 3, to the 
processing region setting unit 15151, continuity setting 
unit 15152, actual world estimating unit 15153, image 
generating unit 15154, and image display unit 15155. 
Further, the signal processing unit 4 causes the image 

20 display unit 15155 to display the input image. 

In step S15212, the user I/F 15156 determines whether 
or not there has been some sort of user input, by the user 
operating the user I/F 15156. In step S15212, in the event 
that determination is made that there is no user input, i.e., 

25 in the event that the user has made no operations, the flow 
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skips steps S15213 through S15215, and proceeds to step 
S15216 . 

On the other hand, in step S15212, in the event that 
determination is made that there has been user input, i.e., 
5 in the event that the user has viewed the input image 

displayed on the image display unit 15155 and operated the 
user I/F 15156, thereby making user input indicating some 
sort of instruction or information, and in the event that 
there has been some sort of user input, the flow proceeds to 
10 step S15213, where the user I/F 15156 determines whether or 
not the user input is user instructions instructing ending 
of the processing of the signal processing device 4 . 

In the event that determination is made that in step 
S15213 that the user input is ending instructions, the 
15 signal processing device 4 ends the processing. 

On the other hand, in the event that determination is 
made in step S15213 that the user input is not ending 
instructions, the flow proceeds to step S15214, where the 
user I/F 15156 determines whether or not the user input is 
2 0 constraint condition information or weighting. In the event 
that determination is made in step S15214 that the user 
input is not constraint condition information nor weighting, 
step S15215 is skipped, and the flow proceeds to step S15216. 
Also, in step S15214, in the event that the user input 
25 is determined to be constraint condition information or 
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weighting, the flow proceeds to step S15215, and the user 
I/F 15156 supplies the constraint condition information or 
weighting to the equation generating unit 15162, and the 
flow proceeds to step S15216. With the user I/F 15156, both 
5 the constraint condition information and the weighting can 
be input, and in the event that the user inputs both the 
constraint condition information and the weighting, the user 
I/F 15156 supplies both the constraint condition information 
and the weighting to the equation generating unit 15162. 

10 In step S15216, the processing region setting unit 

15151 sets the processing region based on the input image, 
and supplies the processing region information identifying 
the processing region to the continuity setting unit 15152, 
the model generating unit 15161 and equation generating unit 

15 15162 of the actual world estimating unit 15153, and image 

generating unit 15154, and the flow proceeds to step S15217. 
Now, an arrangement maybe made wherein the setting of the 
processing region in step S15216 is performed by the user 
inputting processing region instruction information by 

20 operating the user I/F 15156, so as to be carried out based 
on the processing region instruction information, or, this 
may be carried out without processing region instruction 
information from the user. 

In step S15217, the continuity setting unit 15152 

25 recognizes the processing region in the input image, from 
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the processing region information supplied from the 
processing region setting unit 15151. Further, the 
continuity setting unit 15152 sets the continuity of the 
actual world 1 signals of which a portion of the continuity 
5 is lost in the processing region image data, and supplies 
continuity information representing the continuity for the 
processing region to the model generating unit 15161 of the 
actual world estimating unit 15153, and the flow proceeds to 
step S15218. Here, the continuity setting unit 15152 sets 

10 the movement amount v, expressing that the movement is in 

the horizontal direction at a constant speed of v pixels per 
shutter time (exposure time) , as the continuity information, 
and supplies this to the model generating unit 15161 of the 
actual world estimating unit 15153. Here, the continuity 

15 setting unit 15152 sets the movement amount showing only the 
movement size as continuity information, under the premise 
that the object is moving in a horizontal direction, but a 
movement vector showing the size and direction of the 
movement of the object can be set as continuity information 

20 as well. The setting of continuity in step S15217 can be 
arranged by the user inputting the continuity instruction 
information by operating the user I/F 15156, and can be 
performed based on this continuity instruction information, 
or can be performed with no continuity instruction 

25 information from the user. 
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The actual world estimating unit 15153 performs actual 
world estimating processing in step S15218. In other words, 
with the actual world estimating unit 15153, the model 
generating unit 15161 generates an equation (model equation) 
5 as the model (relation model) wherein the relation of the 
pixel values of each of the pixels of the input image which 
has movement blurring and the signal of the actual world 1 
is modeled, based on the movement amount v supplied from the 
continuity setting unit 15152 in step S15217 and the 
10 processing region information supplied from the processing 
region setting unit 15151 in step S15216, and the generated 
model equation is supplied to the equation generating unit 
15162 . 

The equation generating unit 15162 generates a 
15 constraint condition expression based on the constraint 

condition information supplied from the user I/F 15156 in 
step S15215, and sets the weighting supplied from the user 
I/F 15156 into the generated constraint condition expression 
and into the model equation supplied from the model 
20 generating unit 15161 to the equation generating unit 15162. 
Also, the equation generating unit 15162 substitutes the 
pixel values of each of the pixels of the input image into 
an equation formed from the weighted constraint condition 
expression and the model equation, and thus generates an 
25 equation for finding the approximation function as the model 
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(approximation model) wherein the signal of the actual world 
1 is modeled, and supplies this to the actual world waveform 
estimating unit 15163. In the event that the user has not 
specified (set) constraint condition information, that is to 
5 say, in the event that NO is determined to be the case in 
steps S15212 or S15214, the equation generating unit 15162 
performs predetermined processing such as a method for 
solving a simultaneous equation formed of a model equation 
and an equation hypothesizing that the edge portion of the 
10 processing region is "flat", as described in the embodiment 
in Fig. 113. 

Now, the weighting can be set for the model equation 
only, for the constraint condition expression only, or for 
both the constraint condition expression and the model 

15 equation. 

The actual world waveform estimating unit 15163 
estimates the waveform of the actual world 1 signal, that is 
to say, obtains an approximation function approximating the 
actual world 1 signals by solving the equation supplied from 

20 the equation generating unit 15162, and supplies the 

approximation function to the image generating unit 15154, 
as estimation results of the waveform of the actual world 1 
signals . 

The details of the actual world estimating process of 
25 step S15218 will be described later with reference to Fig. 
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150. 

After the processing in step S15218, the flow proceeds 
to step S15219, and the image generating unit 15154 
generates a signal more closely approximating the signal of 
5 the actual world 1, from the approximation function supplied 
from the actual world waveform estimating unit 15163, that 
is to say, generates an image with no movement blurring, and 
based on the processing region information supplied from the 
processing region setting unit 15151, replaces the portion 
10 of the processing region of the input image with an image 

which has no movement blurring generated, generates this as 
an output image, and supplies this to the image display unit 
15155, and the flow proceeds to step S15220. 

In step S15220, the image display unit 15155 displays 
15 the output image supplied from the image generating unit 

15154 instead of the input image displayed in step S15211, 
or along with the input image, and the flow proceeds to step 
S15221 . 

In step S15221, as with the case of step S15212, the 
20 user I/F 15156 determines whether or not there has been some 
sort of user input by the user operating the user I/F 15156, 
and in the event that determination is made that there has 
been no user input, i.e., in the event that the user has 
made no operations, the flow returns to step S15221, and 
25 awaits some sort of user input. 



- 472 - 

S04P0850 

Also, in the event that determination is made in step 
S15221 that there has been user input, i.e., in the event 
that the user has viewed the input image or output image 
displayed on the image display unit 15155 and operated the 
5 user I/F 15156, thereby making user input representing some 
sort of instructions or information, the flow proceeds to 
step S15222, where the user I/F 15156 determines whether or 
not the user input is ending instructions instructing ending 
of the processing of the signal processing device 4 . 

10 In the event that determination is made in step S15222 

that the user input is ending instructions, the signal 
processing device 4 ends the processing. 

On the other hand, in the event that determination is 
made in step S15222 that the user input is not ending 

15 instructions, the flow proceeds to step S15223, where the 
user I/F 15156 determines whether or not the user input is 
constraint condition information or weighting. In the event 
that determination is made in step S15223 that the user 
input is not constraint condition information nor weighting, 

20 the flow returns to step S15221, and hereafter the same 
processing is repeated. 

Also, in the event that determination is made in step 
S15223 that the user input is constraint condition 
information or weighting, the flow returns to step S15215, 

25 and as described above, the user I/F 15156 supplies the 
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constraint condition information or weighting to the 
equation generating unit 15162. Then the flow proceeds from 
step S15215 to S15216, and hereafter the same processing is 
repeated. Now, in the event that the signal processing 
5 device 4 repeats the processing of step S15215 through step 
S15223, the processing of the above-described step S15216 
and S15217 are the same as that set by the first processing 
S15216 and S15217, and the same process as the first process 
may be repeated, or may be skipped. 
10 Thus, with the signal processing device 4 in Fig. 147, 

by finding an approximation function as a model wherein the 
signal of the actual world 1 is modeled, an output image of 
high image quality with no movement blurring can be 
generated. 

15 Also, by operating the user I/F 15156, the user can set 

constraint conditions which constrain the relation between 
each of the pixels in the original image having no movement 
blurring, and in the event that a new constraint condition 
is input, an image without movement blurring can be 

2 0 requested again, and therefore an output image of high image 
quality according to the preference of the user can be 
easily acquired. 

In the signal processing device 4 in Fig. 147, an 
output image without movement blurring is generated by 

2 5 finding an approximation function which approximates the 
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light signal of the actual world 1, but this output image 
can be regarded as having the movement blurring removed from 
the input image. Accordingly, it can be said that 
processing is performed for movement blurring removal with 
5 the signal processing device 4 in Fig. 147. 

Next, the internal configuration of the actual world 
estimating unit 15153 in Fig. 147 will be described with 
reference to Fig. 149. With the embodiment in Fig. 147 also, 
the mechanism for generating movement blurring of objects 

10 reflecting in the input image is modeled as a relation of 
the above -described Expression (141) , which is an example 
shown in Figs. 125 and 126. 

In other words, let us say that the mechanism for 
movement blurring generating is modeled assuming that the 

15 pixel values of each of the pixels of the movement blurring 
image as the input image are values wherein the pixel values 
(or the electric charge corresponding thereto) of each of 
the pixels of an image with no movement blurring which are 
accumulates (integrated) while moving at the movement amount 

20 v. 

The actual world estimating unit 15153 comprises a 
•model generating unit 15161, an equation generating unit 
15162, and an actual world waveform estimating unit 15163. 
Further, the equation generating unit 15162 comprises a 
25 constraint condition setting unit 15171, a weighting 
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changing unit 15172, and a normal equation generating unit 
15173 . 

The movement amount v as the continuity information is 
supplied to the model generating unit 15161 by the 
5 continuity setting unit 15152. The movement amount v in 
this example expresses the continuity of the object moving 
in the horizontal direction at a constant speed of v pixels 
per shutter time, within the processing region of the input 
image which is set by the processing region setting unit 

10 15151, and for example, let us say that v = 4, as with the 

above -described example. Also, the movement amount v can be 
a value which differs for each line, and in the event that 
the movement amount v differs for each line, k number, equal 
to the number of lines, of the movement amount v is supplied 

15 as continuity information from the continuity setting unit 
15152 to the model generating unit 15161. 

Also, processing region information which specifies the 
processing region within the input image is also supplied to 
the model generating unit 15161 from the processing region 

20 setting unit 15151. In the case of using a rectangular- 
shaped region as the processing region, the processing 
region information can be expressed, for example, by 
coordinate values at opposite angles of a rectangular- shaped 
processing region as to an absolute coordinates system 

2 5 wherein a predetermined position (pixel) of the input image 
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is the origin, or the center coordinate of the processing 
region and the width (horizontal direction) and the height 
(vertical direction) of the processing region, or the like. 
The model generating unit 15161 acquires, for example, 
5 a processing region width of 1, and k line numbers, as 
illustrated in Fig. 119, from the processing region 
information supplied from the processing region setting unit 
15151. In this example, let us say that the processing 
region 15042 is rectangular- shaped, and for example, has 1 = 

10 10 and k = 4. In this case, the processing region width of 
each line (the number of pixels) are the same, but with the 
processing region 15042, each line can have a different 
processing region width. In the case that the processing 
region width 1 differs for each line, the model generating 

15 unit 15021 acquires k, which is hew same number as the 
number of lines of the processing region width 1. 

Also, the model generating unit 15161 generates an 
equation (model equation) of a mixed pixel value P± which is 
shown in the above -described Expression (141) , based on the 

2 0 movement amount v supplied from the continuity setting unit 
15152 and the processing region width 1 and k number of 
lines acquired from the processing region information, and 
supplies this to the normal equation generating unit 15173. 
The constraint condition information specified by the 

25 user operating the user I/F 15156 is supplied to the 
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constraint condition setting unit 15171 from the user I/F 
15156. For example, the user performs an operation with the 
user I/F 15156 to select (specify) the desired constraint 
condition from the constraint conditions which constrain the 
5 relation of the pixel values Q h of each of the pixels of an 
image with no movement blurring, such as the above-described 
Expression (143) or Expression (144) , which have been set in 
advance inside the constraint condition setting unit 15171, 
and the user I/F 15156 supplies the information describing 
10 the selected constraint conditions as the constraint 

condition information to the constraint condition setting 
unit 15171. 

The constraint condition setting unit 15171 selects a 
constraint condition expression corresponding to the 

15 constraint condition information supplied from the user I/F 
15156 from the constraint condition expressions within the 
constraint condition setting unit 15171 itself. Further, 
the constraint condition setting unit 15171 acquires the 
processing region width 1 based on the processing region 

20 information of the input image supplied to the equation 
generating unit 15162 from the processing region setting 
unit 15151, and generates only the number which corresponds 
to the processing region width 1, of he constraint condition 
expressions. For example, in the event that Expression 

25 (143) is used which has the condition "adjacent pixel 
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difference = 0" as the constraint condition information, the 
constraint condition setting unit 15171 generates the 
constraint condition expression expressed by the above- 
described Expression (145) , by acquiring the processing 
5 region width 1 based on the processing region information 
supplied from the processing region setting unit 15151, and 
supplies this to the normal equation generating unit 15173. 
Iri the example of the processing region 15042 illustrated in 
Fig. 119, the processing region 15042 is rectangular-shaped, 

10 and the processing region width 1 is the same of each line, 
but in the case that the processing region width 1 differs 
for each line, the number of constraint condition 
expressions also differ for each line. 

The weighting changing unit 15172 has a weighting W mi 

15 (hereafter will be referred to simply as weighting W m i) as to 
the equation for each of the mixed pixel values P± (the 
model equation) and a weighting W b j (hereafter will be 
referred to simply as weighting W bj ) as to the constraint 
condition expression. The values for each of the weighting 

20 W mi and the weighting W b j are supplied from the user I/F 
15156 by the user operating the user I/F 15156, and are 
changed from the default value (for example, 1) . In other 
words, the user can decide (set) each of the values for the 
weighting W mi and the weighting W b j , by operating the user 

25 I/F 15156. The weighting changing unit 15172 supplies each 
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of the weighting W m i and the weighting W b j , set by the user, 
to the normal equation generating unit 15173, and sets 
(changes) the weighting of the expressions corresponding to 
each of the model equation and the constraint condition 
5 expression of the normal equation generating unit 15173. 

Specifically, with the normal equation generating unit 
15173, the weighting W b j is set by the weighting changing 
unit 15172, as to Expression (147) which considers the error 
e b j in the constraint condition Expression (145) which is 
10 supplied from the constraint condition setting unit 15171. 
In other words, Expression (147) is changed to Expression 
(161) which has multiplied both sides thereof by the 
weighting W b j - 

Also, with the normal equation generating unit 15173, 
15 the weighting W mi is set by the weighting changing unit 15172, 
as to Expression (14 6) which considers the error e m i in the 
equation for mixed pixel value Pi (model equation) (141) 
which is supplied from the model generating unit 15161. In 
other words, Expression (146) is changed to the above- 
20 described Expression (162) which has multiplied both sides 
thereof by the weighting W m i . 

The normal equation generating unit 15173 selects one 
predetermined line from the processing region 15042 as the 
line of interest, as illustrated in Fig. 128. Then the 
25 normal equation generating unit 15173 generates the normal 
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equation expressed in the above -described Expression (163) 
from Expression (161) and Expression (162) , corresponding to 
the selected line of interest. 

Here, Expression (163) is replaced with Expression 
5 (164), using the matrix A" and the column vectors x, y, e". 
In this case, the total sum E" of the squared errors of the 
error e b j of Expression (161) and the error e mi of Expression 
(162) can be expressed with Expression (165) , and in order 
to find the Q h which causes the total sum E" of the squared 

10 errors to be the minimum, the column vector x (the column 

vector x with the pixel value Q h as a component thereof) can 
be expressed with Expression (166) , similar to the above- 
described Expressions (150) and (151) . 

Accordingly, the normal equation generating unit 15173 

15 computes the matrix (A" T A" ) ~ 1 A ,|T , which is a coefficient of 
the column vector y of the right side of Expression (166) , 
and substitutes the mixed pixel value Pi which is acquired 
based on the input image supplied from the sensor 2 (Fig. 1) 
into the column vector y. The matrix (A ,,T A ,, )" 1 A ,,T is 

20 computed, and Expression (166) into which the mixed pixel 

value Pi is substituted is supplied from the normal equation 
generating unit 15173 to the actual world waveform 
estimating unit 15163. 

When the processing such as described above is finished 

25 wherein one, predetermined line of the processing region 



- 481 - 

S04P0850 

15042 is selected as the line of interest, the normal 
equation generating unit 15173 selects a line not yet 
selected as the next line of interest, and similar 
processing is performed as to the selected next line of 
5 interest . 

The actual world waveform estimating unit 15163, by 
computing Expression (166) which is supplied from the normal 
equation generating unit 15173, finds the column vector x, 
that is to say, the pixel value Q h with no movement blurring, 
10 and supplies this to the image generating unit 15154 (Fig. 
147) . 

The image generating unit 15154 replaces a portion of 
the processing region 15042 of the input image with the 
image of the pixel value Q h with no movement blurring, which 

15 is supplied from the actual world waveform estimating unit 
15163, based on the processing region information supplied 
from the processing region setting unit 15151, and generates 
this as an output image, and supplies this to the image 
display unit 15155. 

2 0 Next, the actual world estimating processing of the 

actual world estimating unit 15153 in step S15218 in Fig. 
148 will be described with reference to the flowchart in Fig. 
150 . 

First, in step S15251, the model generating unit 15161 
2 5 acquires the processing region width 1 and the number of 
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lines k, based on the processing region information supplied 
from the processing region setting unit 15151, and the flow 
proceeds to step S15252 . 

In step S15252, the model generating unit 15161 
5 acquires k movement amounts v, k being equivalent to the 
number of lines in the processing region 15042, from the 
continuity setting unit 15152, and the flow proceeds to step 
S15253 . 

In step S15253, the model generating unit 15161 

10 generates an equation for the mixed pixel value Pi for each 
line, and the flow proceeds to step S15254. In other words, 
the model generating unit 15161 generates k number of model 
equations (Expression (141) ) . 

In step S15254, the constraint condition setting unit 

15 15171 selects the constraint condition expression 

corresponding to the constraint condition information 
supplied from the user I/F 15156, from the constraint 
condition expressions existing within the constraint 
condition setting unit itself. Also, the constraint 

20 condition setting unit 15171 acquires the processing region 
width 1 from the processing region information of the input 
image which is supplied from the equation generating unit 
15162 from the processing region setting unit 15151, and 
generates an constraint condition Expression (Expression 

25 (145) ) corresponding to the constraint condition information 
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from the user I/F 15156, which comprises the number of 
equations corresponding to the processing region width 1, 
and the flow proceeds to step S15255. 

In step S15255, the weighting changing unit 15172 sets 
5 the weighting of each expression of the mixed pixel value Pi 
which is supplied from the model generating unit 15161 (the 
model equation) and the constraint condition expression 
which is supplied from the constraint condition setting unit 
15171, by supplying the weighting W bj and the weighting W mi 
10 which are supplied from the user I/F 15156 to the normal 
equation generating unit 15173, and the flow proceeds to 
step S15256. 

In step S15256, the normal equation generating unit 
15173 selects one predetermined line from the processing 

15 region 15042 as the line of interest, and the flow proceeds 
to step S15257. In step S15257, the normal equation 
generating unit 15173 generates a normal equation expressed 
by Expression (163) (Expression (164)) from the constraint 
condition Expression (161) wherein the weighting W b j is 

20 changed by the weighting changing unit 15172 as to step 
S15255, in Expression (145) supplied from the constraint 
condition setting unit 15171 and Expression (162) wherein 
the weighting W m i is changed by the weighting changing unit 
15172 in step S12522 as to Expression (146) considering the 

2 5 error e mi of each of the expressions of the mixed pixel value 
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Pi expressed by Expression (141) supplied from the model 
generating unit 15161, and the flow proceeds to step S15258. 

In step S15258, the normal equation generating unit 
15173 computes the matrix (A I|T A" ) " 1 A I|T / which is a 
5 coefficient of the column vector y of the right side of 
Expression (166), and the flow proceeds to step S15259. 

In step S15259, the normal equation generating unit 
15173 substitutes the mixed pixel value Pi which is acquired 
based on the input image supplied from the sensor 2 (Fig. 1) 
10 into the column vector y, and the flow proceeds to step 
S15260. 

In step S15260, the actual world waveform estimating 
unit 15163 computes a normal equation. That is to say, the 
actual world waveform estimating unit 15163 finds a pixel 

15 value Q h with no movement blurring by computing Expression 
(166) which is supplied from the normal equation generating 
unit 15173, and the flow proceeds to step S15261. 

In step S15261, determination is made as to whether or 
not processing is performed to find a pixel value Q h with no 

20 movement blurring as to all lines of the processing region 
15042. In the event that it is determined that processing 
is not performed to find a pixel value Q h with no movement 
blurring as to all lines of the processing region 15042, the 
process is returned to step S15256, and the processing in 

25 steps S15256 through S15261 is repeated. That is to say, a 
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line not yet selected of the processing region 15042 is 
selected as the next line of interest, and processing is 
performed to find a pixel value Q h with no movement blurring. 
On the other hand, in the event that it is determined 
5 that processing is performed to find a pixel value Q h with 
no movement blurring as to all lines of the processing 
region 15042, the actual world estimating processing ends 
and returns . 

Thus, by generating a normal equation (Expression 

10 (163) ) as to one line which is taken as the line of interest 
of the processing region 15042, and repeating the processing 
to find a pixel value Q h with no movement blurring k number 
of times, which is the number of lines in the processing 
region 15042, the pixel value Q h with no movement blurring 

15 can be found across the entire region of the processing 
region 15042 . 

Here, in the event that the movement amount v and 
processing region width 1 for each line of the processing 
region 15042 are each the same, and that the weighting W bj 

20 and the weighting W mi supplied from the user I/F 15156 to the 
weighting changing unit 15172 is set (supplied) similarly 
with each line of the processing region 15042, the matrix A 
of Expression (163) becomes the same for each line. 
Accordingly, the matrix (A" T A" ) _1 A ,|T of Expression (166) 

25 becomes the same for each line, and so the processing for 
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computing the matrix (A" T A" ) " 1 A I|T for each line can be 
omitted. 

Thus, the actual world estimating processing of the 
actual world estimating unit 15153 in step S15218 in Fig. 
5 14 8, in the event that the movement amount v and processing 
region width 1 for each line of the processing region 15042 
are each the same, and that the weighting W b j and the 
weighting W m i supplied from the user I/F 15156 to the 
weighting changing unit 15172 is set (supplied) similarly 

10 with each line of the processing region 15042, will be 

described with reference to the flowchart in Fig. 151. Now, 
with regard to the portions in Fig. 151 which are similar to 
the flowchart in Fig. 150, the description thereof will be 
omitted as appropriate. 

15 In Fig. 151, the processing of step S15288 

corresponding to step S15256 in Fig. 150 is not positioned 
between steps S15285 and S15286, but rather between steps 
S15287 and S15289. Further, in step S15291, in the event 
that it is determined that the processing for finding the 

2 0 pixel value Qh without movement blurring is not performed as 
to all of the lines in the processing region 15042, the 
processing returns the flow to step S15288. 

In other words, in the event that the movement amount v 
and processing region width 1 for each line of the 

25 processing region 15042 are each the same, and that the 
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weighting W b j and the weighting W m i supplied from the user 
I/F 15156 to the weighting changing unit 15172 is set 
(supplied) similarly with each line of the processing region 
15042, the normal equation generated in step S15286 becomes 
5 the same for each line, and therefore, after computing the 
matrix (A" T A M ) _1 A ,,T as to the first line of interest in step 
S15287, computing the matrix (A nT A n ) _1 A n T (the processing in 
step S15287) as to the line of interest hereafter is not 
necessary. Accordingly, the selection of the line of 

10 interest (step S15288) , the substitution of the mixed pixel 
value Pi (the processing in step S15289) , and the computing 
of the normal equation (the processing in step S15290) need 
only be executed in order, as to the lines in the processing 
region 15042 which have not yet been selected. 

15 Thus, with the embodiment illustrated in Fig. 147, a 

normal equation is generated from the equation of the mixed 
pixel value Pi wherein the pixel value with movement 
blurring occurring is modeled with a pixel value with no 
movement blurring, and the constraint condition expression 

20 using the relation between the adjacent pixels of the pixel 
values with no movement blurring, and by computing this 
normal equation, a pixel value with no movement blurring can 
be found, and thus an image with good image quality can be 
generated . 

25 Also, the weighting for each of the mixed pixel value Pi 
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equation and constraint condition expression is supplied to 
the signal processing device 4 and set as desired values by 
the user operating the user I/F 15156, and therefore, an 
image of high image quality according to the preference of 
the user can be generated. 

In the above -described example, the information showing 
which constraint condition the user selected (specified) 
from the several constraint conditions previously set inside 
the constraint condition setting unit 15171 is used as the 
constraint condition information supplied from the user I/F 
15156 to the constraint condition setting unit 15171, but 
the user may also directly input the constraint condition 
expression by operating the user I/F 15156. 

Also, in addition to the above-described embodiment, 
with the constraint condition setting unit 15171, the 
predetermined constraint conditions are determined (fixed) , 
the user can operate the user I/F 15156 to select the 
constraint conditions, or use an embodiment wherein directly 
inputting the constraint condition expression is not 
necessary. 

Now, with the continuity setting unit 15012 in Fig. 113 
(similar with other continuity setting units) , the movement 
as the continuity information can be set based on the user 
operation, but also can be set by detecting the movement 
from the input image . 
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The detecting method of the movement with the 
continuity setting unit 15012 in Fig. 113 will be described. 

In the event that a certain object is moving in an 
input image, as for a method for detecting movement vector 
5 serving as the movement of the object for example, the so- 
called block matching method has been known. 

However, with the block matching method, matching is 
performed between the frame of interest and the frames 
before and after the frame of interest, so that movements 
10 cannot be detected easily with the frame of interest alone. 

To this end, the continuity setting unit 15012 is 
configured so as to detect movements from an input image 
having one frame alone . 

Fig. 152 illustrates a configuration example of the 
15 continuity setting unit 15012 in Fig. 113. 

With the continuity setting unit 15012 of which the 
configuration is shown in Fig. 152, the movement direction 
of an object in the processing region of the an input image 
is detected, and the input image is corrected such that the 
20 movement direction becomes the horizontal direction. 

Subsequently, the features subjected to the one-dimensional 
differentiation in the movement direction of the object in 
the input image, which are the differential value of the 
pixel values of pixels adjacent in the movement direction, 
25 are detected. 
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Further, the correlation is detected between the 
features of the pixel of interest and the features of the 
corresponding pixel with a predetermined distance in the 
movement direction, and the movement amount of the object is 
5 detected according to the distance between the corresponding 
pixel and the pixel of interest, which exhibits the maximum 
detected correlation. 

The continuity setting unit 15012 of which the 
configuration is shown in Fig. 152 includes a movement 
10 direction detecting unit 11201, a movement direction 

correcting unit 11202, a features detecting unit 11203, and 
a movement amount detecting unit 11204. 

Further, the movement direction detecting unit 112 01 
includes an activity computing unit 11211, and an activity 
15 evaluating unit 11212. The movement direction correcting 
unit 11202 includes an affine transformation unit 11213. 

The features detecting unit 11203 includes a difference 
computing unit 11214, a difference evaluating unit 11215, an 
intermediate image creating unit 11216, an intermediate 
20 image creating unit 11217, frame memory 11218, a sign 
inverting unit 11219, and frame memory 11220. 

Further, the movement amount detecting unit 112 04 
includes a correlation detecting unit 11221, and a 
correlation evaluating unit 11222. 
25 With the continuity setting unit 15012 wherein the 
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configuration is illustrated is Fig. 152, the input image is 
supplied to the movement direction detecting unit 11201 and 
the movement direction correcting unit 11202. Further, the 
processing region information that is output from the 
processing region setting unit 15011 in Fig. 113 is also 
supplied to the movement direction detecting unit 11201 and 
the movement direction correcting unit 11202. 

The movement direction detecting unit 112 01 acquires 
the input image and the processing region information, and 
detects the movement direction in the processing region from 
the acquired input image. 

When capturing a moving object, movement blurring 
occurs on the image of the object. This is caused by 
actions of the image sensor of a camera or video camera 
serving as the sensor 2 for capturing the image of an object. 

That is to say, an image sensor such as a CCD (Charge 
Coupled Device) or CMOS (Complementary Metal -Oxide 
Semiconductor) sensor consecutively converts the incident 
light into electric charge for each pixel during exposure 
time (shutter time) , and further converts the electric 
charge into one pixel value. When an object to be captured 
is in a stationary state, the image (light) of the same 
portion of the object is converted into one pixel value 
during exposure time. The image thus captured includes no 
movement blurring . 
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On the other hand, when an object is moving, the image 
of the portion of the object to be cast into one pixel 
changes during exposure time, and the images of different 
portions of the object are converted into one pixel value by 
5 accident. Speaking inversely, the image of one portion of 
the object is cast into multiple pixel values, which is 

it 

movement blurring . 

Movement blurring occurs in the movement direction of 
the object. 

10 Upon focusing on the pixel values of pixels arrayed in 

the movement direction of the portion where movement 
blurring occurs (region including movement blurring) , the 
image of generally the same range portion of the object is 
projected to the pixel values of the pixels arrayed in the 

15 movement direction. Accordingly, we can say that change in 
the pixel values of the pixels arrayed in the movement 
direction at the portion where movement blurring occurs is 
further reduced. 

The movement direction detecting unit 11201 detects a 

20 movement direction based on such change, i.e., activity in 
the pixel value of a pixel in the processing region of an 
input image . 

More specifically, the activity computing unit 11211 of 
the movement direction detecting unit 11201 computes change 
25 (activity) in the pixel values of pixels arrayed in various 



- 493 - 

S04P0850 

directions for each predetermined direction. For example, 
the activity computing unit 11211 computes the difference 
between the pixel values of the pixels positioned 
corresponding to each direction for each predetermine 
5 direction as activity. The activity computing unit 11211 

supplies information indicating change in the computed pixel 
values to the activity evaluating unit 11212. 

The activity evaluating unit 11212 selects the minimum 
change in the pixel value, of change in the pixel values of 

10 the pixels for each predetermined direction supplied from 
the activity computing unit 11211, and takes the direction 
corresponding to the selected change in the pixel value as 
the movement direction. 

The movement direction detecting unit 11201 supplies 

15 movement direction information indicating the movement 

direction thus detected to the movement direction correcting 
unit 11202. 

The movement direction correcting unit 11202 is 
supplied with processing region information as well. The 

20 movement direction correcting unit 11202 converts the image 
data within the processing region in the input image based 
on the movement direction information supplied from the 
movement direction detecting unit 11201 such that the 
movement direction becomes the horizontal direction of the 

2 5 image. 
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For example, the affine transformation unit 11213 of 
the movement direction correcting unit 112 02 subjects the 
image data within the processing region in the input image 
to affine transformation based on the movement direction 
5 information supplied from the movement direction detecting 
unit 11201 such that the movement direction shown in the 

<J 

movement direction information becomes the horizontal 
direction of the image. 

The movement direction correcting unit 11202 supplies 
10 the image data within the processing region in the input 

image converted such that the movement direction becomes the 
horizontal direction of the image to the features detecting 
unit 11203. 

Now, with the input image, the hypothesis is that the 
15 object is moving in the horizontal direction (from the left 
to the right direction) , and therefore the continuity 
setting unit 15012 can be configured without providing the 
movement direction detecting unit 11201 and the movement 
direction correcting unit 11202. 
20 The features detecting unit 11203 detects the features 

of the image supplied from the movement direction correcting 
unit 11202. 

That is to say, the difference computing unit 11214 of 
the features detecting unit 11203 sets a pixel selected by 
25 selecting one pixel from the pixels in the processing region 
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of the input image, as a pixel of interest. Subsequently, 
the difference computing unit 11214 of the features 
detecting unit 112 03 obtains a difference value by 
subtracting the pixel value of the pixel adjacent to the 
5 pixel of interest on the right side from the pixel value of 
the pixel of interest. 

The difference computing unit 11214 obtains difference 
values by taking the pixels in the processing region of the 
input image as a pixel of interest in order. That is to say, 

10 the difference computing unit 11214 obtains difference 

values regarding all of the pixels in the processing region 
of the input image. The difference computing unit 11214 
supplies the difference values thus computed to the 
difference evaluating unit 11215 along with information 

15 indicating the position of the pixel of interest 

corresponding to each obtained difference value (positional 
information indicating the position on the screen of each 
difference value) . 

The difference evaluating unit 11215 determines 

2 0 regarding whether or not the difference values are 0 or more, 
supplies the difference values equal to or greater than 0 to 
the intermediate image creating unit 11216 along with the 
positional information indicating the position on the screen 
of each difference value, and supplies the difference values 

25 less than 0 to the intermediate image creating unit 11217 
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along with the positional information indicating the 
position on the screen of each difference value. 

The intermediate image creating unit 11216 creates, 
based on the difference values equal to or greater than 0 
5 supplied from the difference evaluating unit 11215 along 
with the positional information indicating the position on 
the screen of the difference values, an intermediate image 
made up of each difference value. That is to say, the 
intermediate image creating unit 11216 creates an 

10 intermediate image by setting the difference values equal to 
or greater than 0 supplied from the difference evaluating 
unit 11215 to the pixels at the positions on the screen 
indicated by the positional information, and setting 0 to 
the pixels at the positions where no difference value is 

15 supplied from the difference evaluating unit 11215. The 
intermediate image creating unit 11216 supplies the 
intermediate image thus created (hereafter, referred to as 
non-inverted intermediate image) to the frame memory 11218. 
The intermediate image creating unit 11217 creates, 

20 based on the difference values less than 0 (negative values) 
supplied from the difference evaluating unit 11215 along 
with the positional information indicating the positions on 
the screen of the difference values, an intermediate image 
made up of the difference values. That is to say, the 

25 intermediate image creating unit 11217 creates an 
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intermediate image by setting the difference values less 
than 0 supplied from the difference evaluating unit 11215 to 
the pixels at the positions on the screen indicated by the 
positional information, and setting 0 to the pixels at the 
5 positions where no difference value is supplied from the 
difference evaluating unit 11215. The intermediate image 
creating unit 11216 supplies the intermediate image thus 
created to the sign inverting unit 11219. 

The sign inverting unit 11219 inverts the signs of the 

10 difference values less than 0 set to the pixels of the 
intermediate image supplied from the intermediate image 
creating unit 11217. The signs of the value 0 set to the 
pixels of the intermediate image are not inverted. That is 
to say, the sign inverting unit 11219 selects the difference 

15 values less than 0 set to the pixels of the intermediate 
image supplied from the intermediate image creating unit 
11217, and converts the selected difference values less than 
0 into the values greater than 0 having the same absolute 
values as the difference values. For example, the 

20 difference value, which is -15, is converted into 15 by 

inverting the sign thereof. The sign inverting unit 11219 
supplies the intermediate image thus sign- inverted 
(hereafter referred to as inverted intermediate image) to 
the frame memory 11220. 

25 The frame memory 11218 supplies the non- inverted 
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intermediate image made up of the difference values equal to 
or greater than 0 and 0 to the movement amount detecting 
unit 11204 as features. The frame memory 11220 supplies the 
inverted intermediate image made up of the difference values 
5 greater than 0 of which the signs are inverted and 0 to the 
movement amount detecting unit 11204 as features. 

The movement detecting unit 112 04 detects movements 
based on the features supplied from the features detecting 
unit 11203. That is to say, the movement detecting unit 

10 11204 detects the correlation between the features of at 

least the pixel of interest, of the pixels of the image of 
the object in the processing region of the input image, and 
the features of the corresponding pixel allocated in the 
movement direction as to the pixel of interest, and detects 

15 the movement amount of the image of the object in the 
processing region of the input image according to the 
detected correlation . 

The correlation detecting unit 11221 of the movement 
amount detecting unit 11204 detects the correlation between 

2 0 the non- inverted intermediate image serving as features 
supplied from the frame memory 11218 of the features 
detecting unit 11203, and the inverted intermediate image 
serving as features supplied from the frame memory 1122 0 of 
the features detecting unit 11203. The correlation 

25 detecting unit 11221 supplies the detected correlation to 
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the correlation evaluating unit 11222. 

More specifically describing, for example, the 
correlation detecting unit 11221 of the movement amount 
detecting unit 11204 moves (shifts) the inverted 
5 intermediate image made up of the difference values of which 
the signs are inverted so as to be greater than 0, and 0 
supplied from the frame memory 112 2 0 of the features 
detecting unit 11203, in the horizontal direction of the 
screen in units of pixel as to the non- inverted intermediate 

10 image made up of the difference values equal to or greater 
than 0, and 0 supplied from the frame memory 11218 of the 
features detecting unit 11203. That is to say, the 
correlation detecting unit 11221 moves the positions on the 
screen of the pixels making up the inverted intermediate 

15 image in the horizontal direction. 

The positional relation on the screen between the 
pixels of the non- inverted intermediate image and the pixels 
of the inverted intermediate image changes by moving (the 
pixels of) the inverted intermediate image in the horizontal 

20 direction on the screen. For example, the corresponding 

pixel of the inverted intermediate image positioned on the 
screen corresponding to the pixel of interest of the non- 
inverted intermediate image before movement results in 
departing from the position corresponding to the pixel of 

2 5 interest of the non- inverted intermediate image by the 
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movement distance after movement. More specifically, when 
the non- inverted intermediate image is moved to the right by 
2 0 pixels, the corresponding pixel of the inverted 
intermediate image departs from the position corresponding 
to the pixel of interest of the non-inverted intermediate 
image to the right by 20 pixels. Speaking inversely, the 
corresponding pixel of the inverted intermediate image 
positioned on the screen corresponding to the pixel of 
interest of the non- inverted intermediate image after 
movement departs from the position corresponding to the 
pixel of interest by the movement distance before movement. 

The correlation detecting unit 11221 computes the 
difference between the pixel values of the pixels 
corresponding to the non- inverted intermediate image and the 
inverted intermediate image moved, and takes the sum of the 
differential absolute values as a correlation value. 

For example, the correlation detecting unit 11221 moves 
(shifts) the inverted intermediate image in the horizontal 
direction of the screen in increments of one pixel in a 
range of 70 pixels in the left direction of the screen 
through 70 pixels in the right direction of the screen as to 
the non-inverted intermediate image, computes the difference 
between the pixel values of the pixels to be positioned at 
the same position on the screen regarding the non- inverted 
intermediate image and the inverted intermediate image moved 
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for each moved position (each movement distance) , and takes 
the sum of the differential absolute values as a correlation 
value . 

For example, when the inverted intermediate image is 
5 moved to the left direction of the screen as to the non- 
inverted intermediate image, the movement distance is 
represented with a negative (minus) . When the inverted 
intermediate image is moved to the right direction of the 
screen as to the non- inverted intermediate image, the 

10 movement distance is represented with a positive (plus) . 

The correlation detecting unit 11221 computes the difference 
between the pixel values of the pixels to be positioned at 
the same position on the screen regarding the non- inverted 
intermediate image and the inverted intermediate image moved 

15 for each movement distance of -70 pixels through +70 pixels, 
and takes the sum of the differential absolute values as a 
correlation value. 

The correlation detecting unit 11221 supplies the 
correlation value corresponding to the movement distance to 

20 the correlation evaluating unit 11222. That is to say, the 
correlation detecting unit 11221 supplies a pair of the 
movement distance and the correlation value to the 
correlation evaluating unit 11222. 

The correlation evaluating unit 11222 detects the 

25 movement amount of the image of the object in the processing 
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region of the input image according to the correlation. 
More specifically, the correlation evaluating unit 11222 
takes, of the correlations supplied from the correlation 
detecting unit 11221, the movement distance corresponding to 
5 the maximum (strongest) correlation as a movement amount. 

For example, the correlation evaluating unit 11222 
selects the minimum value, of the sum of the differential 
absolute values serving as the correlation value supplied 
from the correlation detecting unit 11221, and sets the 
10 movement distance corresponding to the selected minimum 
value to the movement amount . 

The correlation evaluating unit 11222 outputs the 
detected movement amount . 

Fig. 153 through Fig. 155 are diagrams for describing 
15 the principle for detecting movements by the continuity 
setting unit 15012 in Fig. 152. 

Now, let us say that a white foreground object serving 
as an object to be captured is disposed in front of a black 
background object serving as another object to be captured, 
2 0 and is moving from the left side to the right side, and a 
camera having an image sensor such as a CCD or CMOS sensor 
captures the foreground object along with the background 
object with a predetermined exposure time (shutter time) . 
In this case, upon focusing on one frame 9f the image 
25 to be output from the camera, the background object is black, 



- 503 - 

S04P0850 

so that the camera outputs, for example, a pixel value 0 as 
to the background object image. The foreground object is 
white, so that the camera outputs, for example, a pixel 
value 255 as to the foreground object image. Note that here, 
5 let us say that the camera outputs a pixel value in a range 
of 0 through 2 8 - 1 . 

The diagram in the upper side of Fig. 153 is a diagram 
illustrating the pixel values of the image to be output by 
the camera when the foreground object is in a stationary 

10 state at the position at the moment that the shutter of the 
camera opens (moment of starting exposure) . 

The diagram in the lower side of Fig. 153 is a diagram 
illustrating the pixel values of the image to be output by 
the camera when the foreground object is in a stationary 

15 state at the position at the moment that the shutter of the 
camera closes (moment of ending exposure) . 

As shown in Fig. 153, the movement amount of the image 
of the foreground object is a distance wherein the image of 
the foreground object moves from the moment that the shutter 

2 0 of the camera opens until the moment that the shutter of the 
camera closes. 

Fig. 154 is a diagram illustrating the pixel values of 
the image to be output from the camera when the foreground 
object moving in front of the background object is captured 
25 by the camera. The image sensor of the camera consecutively 
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converts the image (light) of the object into electric 
charge during exposure time (shutter time) for each pixel, 
and further converts the electric charge into one pixel 
value, and accordingly, the image of the foreground object 
5 11251 is projected into the pixel values of the multiple 

pixels. The maximum pixel value of the image shown in Fig. 
154 is small as compared with the maximum pixel value of the 
image shown in Fig. 153. 

The slope width of the pixel values shown in Fig. 154 
10 corresponds to the width of the image of the background 
object . 

Upon the difference value as to the adjacent pixel on 
the right side being computed, and set to the pixel 
regarding each pixel of the image shown in Fig. 154, the 
15 image made up of the difference values shown in Fig. 155 is 
obtained . 

That is to say, one pixel is selected from the pixels 
of the image shown in Fig. 154, and set to as a pixel of 
interest to which attention is paid. Subsequently, the 

20 difference value is obtained by subtracting the pixel value 
of the pixel adjacent to the pixel of interest on the right 
side from the pixel value of the pixel of interest. The 
difference value is set to the pixel at the position 
corresponding to the pixel of interest. The pixels of the 

2 5 image shown in Fig. 154 are taken as a pixel of interest in 
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order, and the image made up of the difference values shown 
in Fig. 155 is obtained. 

The difference value having a negative (minus) sign 
emerges on the one-pixel left side as to the position of the 
5 foreground object at the moment that the shutter of the 

camera opens shown in the diagram on the upper side of Fig. 
153, and the difference value having a positive (plus) sign 
emerges on the one-pixel left side as to the position of the 
foreground object at the moment that the shutter of the 
10 camera closes shown in the diagram on the lower side of Fig. 
153 . 

Accordingly, upon matching being performed between a 
value obtained by inverting the sign of the difference value 
having a negative (minus) sign and the difference value 
15 having a positive (plus) sign shown in Fig. 155, the 

movement distance of the value obtained by inverting the 
sign of the difference value having a negative (minus) sign 
is the same as the movement amount on the basis of the 
difference value having a positive (plus) sign when 
2 0 performing matching, for example. 

For example, on the basis of the difference value 
having a positive (plus) sign, the value obtained by 
inverting the sign of the difference value having a negative 
(minus) sign is moved in the horizontal direction, the 
25 correlation between the value obtained by inverting the 
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negative difference value and the positive difference value 
is detected for each movement distance thereof, thereby 
detecting the maximum (strongest) correlation. The movement 
distance when the maximum correlation is detected is the 
5 same as the movement amount . 

More specifically, for example, on the basis of the 
difference value having a positive (plus) sign, the value 
obtained by inverting the sign of the difference value 
having a negative (minus) sign is moved in the horizontal 

10 direction, as the correlation between the value obtained by 
inverting the negative difference value and the positive 
difference value for each movement distance thereof, the 
positive difference value is subtracted from the inverted 
value for each pixel. Subsequently, the minimum value 

15 within the subtracted results, i.e., the maximum correlation 
is detected. The movement distance corresponding to the 
detected maximum correlation is the same as the movement 
amount . 

As described above, the movement amount, which is the 
2 0 amount wherein the image of the object moves, can be 

detected from one frame of the image during exposure time 
(shutter time) . 

That is to say, the difference computing unit 11214 of 
the features detecting unit 112 03 selects one pixel from 
25 pixels in the processing region of the input image, sets 
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this as a pixel of interest, and subtracts the pixel value 
of the pixel adjacent to the pixel of interest on the right 
side from the pixel value of the pixel of interest, thereby 
computing the difference values shown in Fig. 155, for 
5 example. The difference evaluating unit 11215 classifies 
the difference values into the positive difference values 
and the negative difference values based on the signs of the 
difference values . 

The intermediate image creating unit 11216 creates, 

10 from the classified positive difference values, a non- 
inverted intermediate image made up of the positive 
difference values thereof. The intermediate image creating 
unit 11217 creates, from the classified negative difference 
values, an intermediate image made up of the negative 

15 difference values thereof. The sign inverting unit 11219 
creates a non-invertied intermediate image by inverting the 
signs of the negative pixel values of the intermediate image 
made up of the negative difference values. 

The movement amount detecting unit 112 04 obtains the 

2 0 movement distance of the non- inverted intermediate image and 
the inverted intermediate image, which have the strongest 
correlation, and takes the obtained movement distance as the 
movement amount . 

When the features detecting unit 11203 detects the 

2 5 image of the moving object, and detects the features of the 
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image of the moving object, the movement amount detecting 
unit 11204 detects a correlation based on the features, and 
detects the movement amount of the image of the object 
within the input image according to the detected correlation. 
5 Also, when the features amount detecting unit 11203 

selects the pixel of interest to which attention is paid 
from the pixels belonged to the image of the moving object, 
and detects the features of the pixel of interest, the 
movement amount detecting unit 112 04 detects the correlation 

10 between the features of the pixel of interest and the 

features of the corresponding pixel to be allocated in the 
movement direction as to the pixel of interest, and detects 
the movement amount of the image of the object in the 
processing region of the input image according to the 

15 detected correlation. 

Fig. 156 is a flowchart for describing the processing 
for detecting a movement amount by the continuity setting 
unit 15012 in Fig. 152. 

In step S11201, the movement direction detecting unit 

20 11201 and the movement direction correcting unit 11202 
acquire the input image and the processing region 
information, and the flow proceeds to step S11202. 

In step S11202, the activity computing unit 11211 of 
the movement direction detecting unit 112011 computes 

25 activity regarding the pixels of the processing region in 
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the input image acquired in the processing in step S11201, 
and the flow proceeds to step S11203. 

For example, the activity computing unit 11211 selects 
the pixel of interest to which attention is paid, of the 
5 pixels of the processing region in the input image. The 
activity computing unit 11211 extracts a predetermined 
number of perimeter pixels in the vicinity of the pixel of 
interest. For example, the activity computing unit 11211 
extracts perimeter pixels made up of 5 x 5 pixels (5 pixels 

10 by 5 pixels) centered on the pixel of interest. 

Subsequently, the activity computing unit 11211 detects 
activity corresponding to the predetermined direction on the 
image from the extracted perimeter pixels. 

With the following description, a pixel array in the 

15 horizontal direction is referred to as a row, and a pixel 

array in the vertical direction is referred to as a column. 

Regarding 5x5 perimeter pixels for example, the 
activity computing unit 11211 detects activity as to the 
angle of 90 degrees (vertical direction of the screen) on 

20 the basis of the horizontal direction of the screen by 

computing the differences between the pixel values of the 
adjacent pixels in the up-and-down (vertical) direction on 
the screen, dividing the sum of the differential absolute 
values computed by the number of differences, and taking the 

25 result as activity. 
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For example, the difference of pixel values is computed 
regarding the two pixels adjacent in the up- to-down 
direction on the screen of 20 pairs, the sum of the 
differential absolute values computed is divided by 20, and 
5 the result (quotient) is set to the activity as to the angle 
of 90 degrees. 

Regarding 5x5 perimeter pixels for example, the 
activity computing unit 11211 detects activity as to the 
angle of 76 degrees (tan -1 (4/1) ) on the basis of the 

10 horizontal direction of the screen by computing the 

differences between the respective pixel values of the 
leftmost pixel through the fourth pixel from the left in the 
lowest row, and the respective pixel values of the pixels on 
the four-pixel upper side and also on one-pixel right side 

15 as to the respective pixels, dividing the sum of the 

differential absolute values computed by the number of 
differences, and taking the result as activity. 

For example, the difference of pixel values is computed 
regarding the two pixels in the upper right direction having 

2 0 a distance of four pixels in the vertical direction, and one 
pixel in the horizontal direction of four pairs, the sum of 
the differential absolute values computed is divided by four, 
and the result (quotient) is set to the activity as to the 
angle of 76 degrees. 

25 The activity computing unit 11211 detects activity as 
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to the angle in a range of 90 degrees through 180 degrees on 
the basis of the horizontal direction of the screen with the 
same processing. In the event of detecting activity as to 
the angle in a range of 90 degrees through 180 degrees, 
5 activity is calculated based on the difference of the pixel 
values of the pixels positioned in the upper left direction. 

The activity thus detected is taken as the activity as 
to the pixel of interest. 

Note that the detected activity may be the activity as 
10 to the perimeter pixels. 

Also, description has been made that the perimeter 
pixels are made up of 5 x 5 pixels (5 pixels by 5 pixels) , 
but pixels having a desired range may be employed regardless 
of 5 x 5 pixels. In the event of employing a great number 
15 of perimeter pixels, angular resolution improves. 

The activity computing unit 11211 supplies information 
indicating activity corresponding to multiple directions to 
the activity evaluating unit 11212. 

Returning to Fig. 156, in step S11203, the activity 
20 evaluating unit 11212 obtains the movement direction by 
selecting the minimum activity based on the activity 
corresponding to a predetermined direction calculated in the 
processing in step S11202, and taking the selected direction 
as the movement direction, and the flow proceeds to step 
25 S11204. 
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In step S11204, the movement direction correcting unit 
112 02 converts the image data in the processing region of 
the input image based on the movement direction obtained in 
the processing in step S112 03 such that the movement 
5 direction becomes the horizontal direction of the image, and 
5 the flow proceeds to step S11205. For example, in step 

S11204, the affine transformation unit 11213 of the movement 

2 

direction correcting unit 11202 subjects the image data in 
the processing region of the input image to affine 

10 transformation based on the movement direction obtained in 
the processing in step S112 03 such that the movement 
direction becomes the horizontal direction of the image. 
More specifically, for example, the affine transformation 
unit 11213 subjects the image data in the processing region 

15 of the input image to affine transformation so as to rotate 
18 degrees in the clockwise direction on the basis of the 
horizontal direction of the screen when the movement 
direction is the angle of 18 degrees. 

In step S11205, the difference computing unit 11214 of 

20 the features detecting unit 11203 computes the difference 
value of the pixel value as to the pixel adjacent in the 
horizontal direction regarding each pixel in the processing 
region of the input image converted such that the movement 
direction becomes the horizontal direction of the screen in 

25 the processing in step S11204, and the flow proceeds to step 
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S11206 . 

For example, in step S112 05, the difference computing 
unit 11214 sets a pixel of interest to which attention is 
paid by selecting one pixel from the pixels in the 
5 processing region of the input image. Subsequently, the 

difference computing unit 11214 obtains the difference value 
by subtracting the pixel value of the pixel adjacent to the 
pixel of interest on the right side from the pixel value of 
the pixel of interest. 

10 In step S11206, the difference evaluating unit 11215 of 

the features detecting unit 11203 allocates the difference 
values based on the signs of the difference values, and the 
flow proceeds to step S11207. That is to say, the 
difference evaluating unit 11215 supplies the difference 

15 values equal to or greater than 0 to the intermediate image 
creating unit 11216, and supplies the difference values less 
than 0 to the intermediate image creating unit 11217. In 
this case, the difference evaluating unit 11215 supplies the 
difference values to the intermediate image creating unit 

20 11216 or the intermediate image creating unit 11217 along 
with the positional information indicating the position of 
each difference value on the screen. 

In step S11207, the intermediate image creating unit 
11216 of the features detecting unit 11203 creates an 

2 5 intermediate image made up of the positive difference values 
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based on the difference values equal to or greater than 0 
(positive difference values) allocated in the processing in 
step S11206, and the flow proceeds to step S11208. That is 
to say, in step S112 07, the intermediate image creating unit 
5 11216 creates an intermediate image by setting the positive 
difference values to the pixels at the positions on the 
screen indicated with the positional information, and 
setting 0 to the pixels at the positions where no difference 
value was supplied. 

10 Thus, a non- inverted intermediate image is created in 

the processing in step S11207. In step S11208, the 
intermediate image creating unit 11217 of the features 
detecting unit 11203 creates an intermediate image made up 
of the negative difference values based on the difference 

15 values less than 0 (negative difference values) allocated in 
the processing in step S11206, and the flow proceeds to step 
S11209. That is to say, in step S11208, the intermediate 
image creating unit 11217 creates an intermediate image by 
setting the negative difference values to the pixels at the 

20 positions on the screen indicated with the positional 

information, and setting 0 to the pixels at the positions 
where no difference value was supplied. 

In step S11209, the sign inverting unit 11219 of the 
features detecting unit 11203 inverts the signs of the 

25 negative difference values of the intermediate image made up 
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of the negative difference values created in the processing 
in step S11208. That is to say, in step S11209, the 
negative difference values set to the pixels of the negative 
intermediate image are converted into the positive values 
5 having the same absolute values. 

Thus, in step S11209, a non- inverted intermediate image 
is created, and then the flow proceeds to step S11210. 

In step S11210, the movement amount detecting unit 
11204 executes correlation detecting processing. The 

10 details of the processing in step S11210 will be described 
later with reference to the flowchart in Fig. 157. 

In step S11211, the correlation evaluating unit 11222 
selects the strongest correlation, of the correlations 
detected in the processing in step S11210, and the flow 

15 proceeds to step S11212. For example, in step S11211, of 
the correlation values serving as the sum of the 
differential absolute values of the pixel values, the 
minimum correlation value is selected. 

In step S11212, the correlation evaluating unit 11222 

2 0 sets the movement distance corresponding to the strongest 

correlation selected in the processing in step S11211 to the 
movement amount, and the flow proceeds to step S11213. For 
example, in step S11212, of the correlation values serving 
as the sum of the differential absolute values of the pixel 

2 5 values, the movement distance of the inverted intermediate 
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image stored in the processing in step S11223 described 
later corresponding to the selected minimum correlation 
value is set to the movement amount. 

In step S11213, the movement amount detecting unit 
5 112 04 outputs the movement amount detected in the processing 
in step S11210, and the processing ends. 

Fig. 157 is a flowchart for describing correlation 
detecting processing corresponding to the processing in step 
S11210 . 

10 In step S11221, the correlation detecting unit 11221 of 

the movement amount detecting unit 11204 moves the positions 
of the pixels of the inverted intermediate image created in 
the processing in step S11209 in the horizontal direction in 
increments of pixel, and the flow proceeds to step S11222. 

15 In step S11222, the correlation detecting unit 11221 

detects the correlation between the non-inverted 
intermediate image and the inverted intermediate image of 
which the positions of the pixels are moved in the 
processing in step S11221, and the flow proceeds to step 

20 S11223. For example, in step S11222, the differences are 

computed between the pixel values of the pixels of the non- 
inverted intermediate image and the pixel values of the 
pixels of the inverted intermediate image at the 
corresponding positions on the screen, and the sum of the 

25 computed differential absolute values is detected as a 
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correlation value. The correlation detecting unit 11221 
supplies correlation information indicating the detected 
correlation to the correlation evaluating unit 11222 along 
with the movement distance of the pixels of the inverted 
5 intermediate image in the processing in step S11221. 

In step S11223, the correlation evaluating unit 11222 
stores the correlation detected in the processing in step 
S11222 along with the movement distance of the pixels of the 
inverted intermediate image in the processing in step S11221, 

10 and the flow proceeds to step S11224. For example, the 
correlation evaluating unit 11222 stores the correlation 
value serving as the sum of the differential absolute values 
of the pixel values along with the movement distance of the 
pixels of the inverted intermediate image in the processing 

15 in step S11221. 

In step S11224, the correlation detecting unit 11221 
determines regarding whether or not correlation as to all of 
the movement distances has been detected, and in the event 
that determination is made that correlation has not been 

20 detected with some movement distances, the flow returns to 
step S11221, where the processing for detecting correlation 
as to the next movement distance is repeated. 

For example, in step S11224, the correlation detecting 
unit 11221 determines whether or not all correlations have 

25 been detected in a case of moving pixels of the inverted 
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intermediate image in the range of 70 pixels in the left 
direction in the image through 70 pixels in the right 
direction in the image. 

In the event that determination is made in step S11224 
5 that the correlations for all movements amount have been 
detected, the processing ends (returns) . 

Thus, the correlation detecting unit 11221 can detect 
correlation. 

As described above, the continuity setting unit 15012 
10 of which the configuration is shown in Fig. 152 can detect 
movement amount from one frame in an image . 

Now, while movement has been detected here with regard 
to a processing region, an arrangement may be made wherein 
movement over the entire screen due to hand shaking, for 
15 example, can be detected, by processing the entire screen. 

Also, even in the event that there are a great amount 
of repetitive patterns of the same design in the input image, 
these can be detected in an accurate manner so long as the 
movement distance and movement direction of the processing 
20 region in the input image to be processed is constant. 

While movement amount detection from one frame in an 
image has been described above, it is needless to say that 
an arrangement may be made wherein movement amount is 
detected from one field. 
2 5 Also, an arrangement may be made wherein the movement 
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amount is detected regarding only the perimeter of the 
selected pixel of interest. 

The above-described embodiments include, besides 
embodiments of the invention described in the Claims, 
5 embodiments regarding the first and second signal processing 
devices given below. 

A first signal processing device comprises: processing 
region setting means for setting the processing region 
within data, wherein light signals of the real world have 

10 been projected on a plurality of pixels each having time 

integration effects and a part of the continuity of the real 
world light signals has been lost; movement vector setting 
means for setting movement vectors of an object within the 
image data corresponding to the continuity of the real world 

15 light signals of which a part of the continuity has been 

lost in the image data; model generating means for modeling 
the region between pixel values of each of the pixels within 
the processing region and pixel values of each of the pixels 
where there is no movement blurring, with the understanding 

20 that the pixel values of each of the pixels within the 

processing region are values obtained by pixel values of 
each of the pixels without movement blurring as to the 
object being integrated while moving corresponding to a 
movement vector; normal equation generating means for 

25 generating normal equations from a first equation wherein 
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the pixel values of the pixels within the processing region 
have been substituted into a model generated by the model 
generating means, and a second equation for constraining the 
relation between the pixels where there is no movement 
blurring; activity detecting means for detecting the 
activity corresponding to each of the pixels within the 
processing region; weight changing means for changing the 
weighting as to a part of the first and second equations 
according to the detection results by the activity detecting 
means; and actual world estimating means for estimating the 
pixel values of each of the pixels where there is no 
movement blurring, by computing the normal equations 
generated by the normal equation generating means and of 
which the weighting has been changed by the weight changing 
means . 

With the second signal processing device, in addition 
to the features of the first signal processing device, the 
weighting changing means change the weighting as to a 
portion of the first and second equations, corresponding to 
the pixel of interest, to zero, when the activity 
corresponding to the pixel of interest within the processing 
region of the image data is greater than the predetermined 
threshold value. 

The above -described embodiments also include 
embodiments regarding signal processing methods, programs, 
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and recording mediums, given below. 

A signal processing method, program, and recording 
medium, comprise: a processing region setting step for 
setting the processing region within data, wherein light 
signals of the real world have been projected on a plurality 
of pixels each having time integration effects and a part of 
the continuity of the real world light signals has been 
lost; a movement vector setting step for setting movement 
vectors of an object within the image data corresponding to 
the continuity of the real world light signals of which a 
part of the continuity has been lost in the image data; a 
model generating step for modeling the region between pixel 
values of each of the pixels within the processing region 
and pixel values of each of the pixels where there is no 
movement blurring, with the understanding that the pixel 
values of each of the pixels within the processing region 
are values obtained by pixel values of each of the pixels 
without movement blurring as to the object being integrated 
while moving corresponding to a movement vector; a normal 
equation generating step for generating normal equations 
from a first equation wherein the pixel values of the pixels 
within the processing region have been substituted into a 
model generated in the model generating step, and a second 
equation for constraining the relation between the pixels 
where there is no movement blurring; an activity detecting 
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step for detecting the activity corresponding to each of the 
pixels within the processing region; a weight changing step 
for changing the weighting as to a part of the first and 
second equations according to the detection results in the 
activity detecting step; and an actual world estimating step 
for estimating the pixel values of each of the pixels where 
there is no movement blurring, by computing the normal 
equations generated by the normal equation generating step 
and of which the weighting has been changed in the weight 
changing step. 
Industrial Applicability 

As described above, according to the present invention, 
images and the like closer approximating real world signals 
can be obtained. 



